Original author(s) | Douglas McIlroy (AT&T Bell Laboratories) |
---|---|
Developer(s) | Various open-source and commercial developers |
Initial release | June 1974 |
Written in | C |
Operating system | Unix, Unix-like, V, Plan 9, Inferno |
Platform | Cross-platform |
Type | Command |
License | Plan 9: MIT License |
In computing, the utility diff is a data comparison tool that computes and displays the differences between the contents of files. Unlike edit distance notions used for other purposes, diff is line-oriented rather than character-oriented, but it is like Levenshtein distance in that it tries to determine the smallest set of deletions and insertions to create one file from the other. The utility displays the changes in one of several standard formats, such that both humans or computers can parse the changes, and use them for patching.
Typically, diff is used to show the changes between two versions of the same file. Modern implementations also support binary files.[1] The output is called a "diff", or a patch, since the output can be applied with the Unix program patch. The output of similar file comparison utilities is also called a "diff"; like the use of the word "grep" for describing the act of searching, the word diff became a generic term for calculating data difference and the results thereof.[2] The POSIX standard specifies the behavior of the "diff" and "patch" utilities and their file formats.[3]
History
diff was developed in the early 1970s on the Unix operating system, which was emerging from Bell Labs in Murray Hill, New Jersey. The first released version shipped with the 5th Edition of Unix in 1974[citation needed], and was written by Douglas McIlroy, and James Hunt. This research was published in a 1976 paper co-written with James W. Hunt, who developed an initial prototype of diff.[4] The algorithm this paper described became known as the Hunt–Szymanski algorithm.
McIlroy's work was preceded and influenced by Steve Johnson's comparison program on GECOS and Mike Lesk's proof program. Proof also originated on Unix and, like diff, produced line-by-line changes and even used angle-brackets (">" and "<") for presenting line insertions and deletions in the program's output. The heuristics used in these early applications were, however, deemed unreliable. The potential usefulness of a diff tool provoked McIlroy into researching and designing a more robust tool that could be used in a variety of tasks, but perform well in the processing and size limitations of the PDP-11's hardware. His approach to the problem resulted from collaboration with individuals at Bell Labs including Alfred Aho, Elliot Pinson, Jeffrey Ullman, and Harold S. Stone.
In the context of Unix, the use of the ed line editor provided diff with the natural ability to create machine-usable "edit scripts". These edit scripts, when saved to a file, can, along with the original file, be reconstituted by ed into the modified file in its entirety. This greatly reduced the secondary storage necessary to maintain multiple versions of a file. McIlroy considered writing a post-processor for diff where a variety of output formats could be designed and implemented, but he found it more frugal and simpler to have diff be responsible for generating the syntax and reverse-order input accepted by the ed command.
Late in 1984 Larry Wall created a separate utility, patch, releasing its source code on the mod.sources and net.sources newsgroups.[5][6][7] This program generalized and extended the ability to modify files with output from diff.
Modes in Emacs also allow for converting the format of patches and even editing patches interactively.
In diff's early years, common uses included comparing changes in the source of software code and markup for technical documents, verifying program debugging output, comparing filesystem listings and analyzing computer assembly code. The output targeted for ed was motivated to provide compression for a sequence of modifications made to a file. The Source Code Control System (SCCS) and its ability to archive revisions emerged in the late 1970s as a consequence of storing edit scripts from diff.
Algorithm
The operation of diff is based on solving the longest common subsequence problem.[4]
In this problem, given two sequences of items:
a b c d f g h j q z
a b c d e f g i j k r x y z
and we want to find a longest sequence of items that is present in both original sequences in the same order. That is, we want to find a new sequence which can be obtained from the first original sequence by deleting some items, and from the second original sequence by deleting other items. We also want this sequence to be as long as possible. In this case it is
a b c d f g j z
From a longest common subsequence it is only a small step to get diff-like output: if an item is absent in the subsequence but present in the first original sequence, it must have been deleted (as indicated by the '-' marks, below). If it is absent in the subsequence but present in the second original sequence, it must have been inserted (as indicated by the '+' marks).
e h i q k r x y + - + - + + + +
Usage
The diff
command is invoked from the command line, passing it the names of two files: diff original new
. The output of the command represents the changes required to transform the original file into the new file.
If original and new are directories, then diff will be run on each file that exists in both directories. An option, -r
, will recursively descend any matching subdirectories to compare files between directories.
Any of the examples in the article use the following two files, original and new:
original: This part of the
document has stayed the
same from version to
version. It shouldn't
be shown if it doesn't
change. Otherwise, that
would not be helping to
compress the size of the
changes.
This paragraph contains
text that is outdated.
It will be deleted in the
near future.
It is important to spell
check this dokument. On
the other hand, a
misspelled word isn't
the end of the world.
Nothing in the rest of
this paragraph needs to
be changed. Things can
be added after it.
|
new: This is an important
notice! It should
therefore be located at
the beginning of this
document!
This part of the
document has stayed the
same from version to
version. It shouldn't
be shown if it doesn't
change. Otherwise, that
would not be helping to
compress the size of the
changes.
It is important to spell
check this document. On
the other hand, a
misspelled word isn't
the end of the world.
Nothing in the rest of
this paragraph needs to
be changed. Things can
be added after it.
This paragraph contains
important new additions
to this document.
|
The command 0a1,6 > This is an important > notice! It should > therefore be located at > the beginning of this > document! > 11,15d16 < This paragraph contains < text that is outdated. < It will be deleted in the < near future. < 17c18 < check this dokument. On --- > check this document. On 24a26,29 > > This paragraph contains > important new additions > to this document.
|
In this traditional output format, a stands for added, d for deleted and c for changed. Line numbers of the original file appear before a/d/c and those of the new file appear after. The less-than and greater-than signs (at the beginning of lines that are added, deleted or changed) indicate which file the lines appear in. Addition lines are added to the original file to appear in the new file. Deletion lines are deleted from the original file to be missing in the new file.
By default, lines common to both files are not shown. Lines that have moved are shown as added at their new location and as deleted from their old location.[8] However, some diff tools highlight moved lines.
Output variations
Edit script
An ed script can still be generated by modern versions of diff with the -e
option. The resulting edit script for this example is as follows:
24a This paragraph contains important new additions to this document. . 17c check this document. On . 11,15d 0a This is an important notice! It should therefore be located at the beginning of this document! .
In order to transform the content of file original into the content of file new using ed, we should append two lines to this diff file, one line containing a w
(write) command, and one containing a q
(quit) command (e.g. by printf "w\nq\n" >> mydiff
). Here we gave the diff file the name mydiff and the transformation will then happen when we run ed -s original < mydiff
.
Context format
The Berkeley distribution of Unix made a point of adding the context format (-c
) and the ability to recurse on filesystem directory structures (-r
), adding those features in 2.8 BSD, released in July 1981. The context format of diff introduced at Berkeley helped with distributing patches for source code that may have been changed minimally.
In the context format, any changed lines are shown alongside unchanged lines before and after. The inclusion of any number of unchanged lines provides a context to the patch. The context consists of lines that have not changed between the two files and serve as a reference to locate the lines' place in a modified file and find the intended location for a change to be applied regardless of whether the line numbers still correspond. The context format introduces greater readability for humans and reliability when applying the patch, and an output which is accepted as input to the patch program. This intelligent behavior isn't possible with the traditional diff output.
The number of unchanged lines shown above and below a change hunk can be defined by the user, even zero, but three lines is typically the default. If the context of unchanged lines in a hunk overlap with an adjacent hunk, then diff will avoid duplicating the unchanged lines and merge the hunks into a single hunk.
A "!
" represents a change between lines that correspond in the two files, whereas a "+
" represents the addition of a line, and a "-
" the removal of a line. A blank space represents an unchanged line. At the beginning of the patch is the file information, including the full path and a time stamp delimited by a tab character. At the beginning of each hunk are the line numbers that apply for the corresponding change in the files. A number range appearing between sets of three asterisks applies to the original file, while sets of three dashes apply to the new file. The hunk ranges specify the starting and ending line numbers in the respective file.
The command diff -c original new
produces the following output:
*** /path/to/original timestamp
--- /path/to/new timestamp
***************
*** 1,3 ****
--- 1,9 ----
+ This is an important
+ notice! It should
+ therefore be located at
+ the beginning of this
+ document!
+
This part of the
document has stayed the
same from version to
***************
*** 8,20 ****
compress the size of the
changes.
- This paragraph contains
- text that is outdated.
- It will be deleted in the
- near future.
It is important to spell
! check this dokument. On
the other hand, a
misspelled word isn't
the end of the world.
--- 14,21 ----
compress the size of the
changes.
It is important to spell
! check this document. On
the other hand, a
misspelled word isn't
the end of the world.
***************
*** 22,24 ****
--- 23,29 ----
this paragraph needs to
be changed. Things can
be added after it.
+
+ This paragraph contains
+ important new additions
+ to this document.
Note: Here, the diff output is shown with colors to make it easier to read. The diff utility does not produce colored output; its output is plain text. However, many tools can show the output with colors by using syntax highlighting.
Unified format
The unified format (or unidiff)[9][10] inherits the technical improvements made by the context format, but produces a smaller diff with old and new text presented immediately adjacent. Unified format is usually invoked using the "-u
" command line option. This output is often used as input to the patch program. Many projects specifically request that "diffs" be submitted in the unified format, making unified diff format the most common format for exchange between software developers.
Unified context diffs were originally developed by Wayne Davison in August 1990 (in unidiff which appeared in Volume 14 of comp.sources.misc). Richard Stallman added unified diff support to the GNU Project's diff utility one month later, and the feature debuted in GNU diff 1.15, released in January 1991. GNU diff has since generalized the context format to allow arbitrary formatting of diffs.
The format starts with the same two-line header as the context format, except that the original file is preceded by "---" and the new file is preceded by "+++". Following this are one or more change hunks that contain the line differences in the file. The unchanged, contextual lines are preceded by a space character, addition lines are preceded by a plus sign, and deletion lines are preceded by a minus sign.
A hunk begins with range information and is immediately followed with the line additions, line deletions, and any number of the contextual lines. The range information is surrounded by double at signs, and combines onto a single line what appears on two lines in the context format (above). The format of the range information line is as follows:
@@ -l,s +l,s @@ optional section heading
The hunk range information contains two hunk ranges. The range for the hunk of the original file is preceded by a minus symbol, and the range for the new file is preceded by a plus symbol. Each hunk range is of the format l,s where l is the starting line number and s is the number of lines the change hunk applies to for each respective file. In many versions of GNU diff, each range can omit the comma and trailing value s, in which case s defaults to 1. Note that the only really interesting value is the l line number of the first range; all the other values can be computed from the diff.
The hunk range for the original should be the sum of all contextual and deletion (including changed) hunk lines. The hunk range for the new file should be a sum of all contextual and addition (including changed) hunk lines. If hunk size information does not correspond with the number of lines in the hunk, then the diff could be considered invalid and be rejected.
Optionally, the hunk range can be followed by the heading of the section or function that the hunk is part of. This is mainly useful to make the diff easier to read. When creating a diff with GNU diff, the heading is identified by regular expression matching.[11]
If a line is modified, it is represented as a deletion and addition. Since the hunks of the original and new file appear in the same hunk, such changes would appear adjacent to one another.[12] An occurrence of this in the example below is:
-check this dokument. On +check this document. On
The command diff -u original new
produces the following output:
--- /path/to/original timestamp
+++ /path/to/new timestamp
@@ -1,3 +1,9 @@
+This is an important
+notice! It should
+therefore be located at
+the beginning of this
+document!
+
This part of the
document has stayed the
same from version to
@@ -8,13 +14,8 @@
compress the size of the
changes.
-This paragraph contains
-text that is outdated.
-It will be deleted in the
-near future.
-
It is important to spell
-check this dokument. On
+check this document. On
the other hand, a
misspelled word isn't
the end of the world.
@@ -22,3 +23,7 @@
this paragraph needs to
be changed. Things can
be added after it.
+
+This paragraph contains
+important new additions
+to this document.
Note: Here, the diff output is shown with colors to make it easier to read. The diff utility does not produce colored output; its output is plain text. However, many tools can show the output with colors by using syntax highlighting.
Note that to successfully separate the file names from the timestamps, the delimiter between them is a tab character. This is invisible on screen and can be lost when diffs are copy/pasted from console/terminal screens.
There are some modifications and extensions to the diff formats that are used and understood by certain programs and in certain contexts. For example, some revision control systems—such as Subversion—specify a version number, "working copy", or any other comment instead of or in addition to a timestamp in the diff's header section.
Some tools allow diffs for several different files to be merged into one, using a header for each modified file that may look something like this:
Index: path/to/file.cpp
The special case of files that do not end in a newline is not handled. Neither the unidiff utility nor the POSIX diff standard define a way to handle this type of files. (Indeed, such files are not "text" files by strict POSIX definitions.[13]) The patch program is not aware even of an implementation specific diff output.
Changes since 1975 include improvements to the core algorithm, the addition of useful features to the command, and the design of new output formats. The basic algorithm is described in the papers An O(ND) Difference Algorithm and its Variations by Eugene W. Myers[14] and in A File Comparison Program by Webb Miller and Myers.[15] The algorithm was independently discovered and described in Algorithms for Approximate String Matching, by Esko Ukkonen.[16] The first editions of the diff program were designed for line comparisons of text files expecting the newline character to delimit lines. By the 1980s, support for binary files resulted in a shift in the application's design and implementation.
GNU diff and diff3 are included in the diffutils package with other diff and patch related utilities.[17] Nowadays there is also a patchutils package that can combine, rearrange, compare and fix context diffs and unified diffs.[18]
Formatters and front-ends
Postprocessors sdiff and diffmk render side-by-side diff listings and applied change marks to printed documents, respectively. Both were developed elsewhere in Bell Labs in or before 1981.[citation needed][discuss]
Diff3 compares one file against two other files by reconciling two diffs. It was originally conceived by Paul Jensen to reconcile changes made by two people editing a common source. It is also used by revision control systems, e.g. RCS, for merging.[19]
Emacs has Ediff for showing the changes a patch would provide in a user interface that combines interactive editing and merging capabilities for patch files.
Vim provides vimdiff to compare from two to eight files, with differences highlighted in color.[20] While historically invoking the diff program, modern vim uses git's fork of xdiff library (LibXDiff) code, providing improved speed and functionality.[21]
GNU Wdiff[22] is a front end to diff that shows the words or phrases that changed in a text document of written language even in the presence of word-wrapping or different column widths.
colordiff is a Perl wrapper for 'diff' and produces the same output but with pretty 'syntax' highlighting.[23]
Algorithmic derivatives
Utilities that compare source files by their syntactic structure have been built mostly as research tools for some programming languages;[24][25][26] some are available as commercial tools.[27][28] In addition, free tools that perform syntax-aware diff include:
- C++: zograscope, AST-based.[29]
- HTML: Daisydiff,[30] html-differ.
- XML: xmldiffpatch by Microsoft and xmldiffmerge for IBM.[31][32]
- JavaScript: astii (AST-based).
- Multi-language: Pretty Diff (format code and then diff)[33]
spiff is a variant of diff that ignores differences in floating point calculations with roundoff errors and whitespace, both of which are generally irrelevant to source code comparison. Bellcore wrote the original version.[34][35] An HPUX port is the most current public release. spiff does not support binary files. spiff outputs to the standard output in standard diff format and accepts inputs in the C, Bourne shell, Fortran, Modula-2 and Lisp programming languages.[36][37][34][38][35]
LibXDiff is an LGPL library that provides an interface to many algorithms from 1998. An improved Myers algorithm with Rabin fingerprint was originally implemented (as of the final release of 2008),[39] but git and libgit2's fork has since expanded the repository with many of its own. One algorithm called "histogram" is generally regarded as much better than the original Myers algorithm, both in speed and quality.[40][41] This is the modern version of LibXDiff used by Vim.[21]
See also
Other free file comparison tools
References
- ^ MacKenzie et al. "Binary Files and Forcing Text Comparison" in Comparing and Merging Files with GNU Diff and Patch. Downloaded 28 April 2007. [1] Archived 2017-12-19 at the Wayback Machine
- ^ Eric S. Raymond (ed.), "diff" Archived 2014-01-31 at the Wayback Machine, The Jargon File, version 4.4.7
- ^ IEEE Computer Society; The Open Group (26 September 2008). Standard for Information Technology—Portable Operating System Interface (POSIX) Base Specifications, Issue 7. pp. 2599–2607. IEEE Std. 1003.1-2001 specifies traditional, "ed script", and context diff output formats; IEEE Std. 1003.1-2008 added the (by then more common) unified format.
- ^ a b James W. Hunt; M. Douglas McIlroy (June 1976). "An Algorithm for Differential File Comparison" (PDF). Computing Science Technical Report, Bell Laboratories. 41. Archived (PDF) from the original on 2014-12-26. Retrieved 2015-05-06.
- ^ Larry Wall (November 9, 1984). "A patch applier--YOU WANT THIS!!!". Newsgroup: net.sources. Usenet: 1457@sdcrdcf.UUCP. Archived from the original on February 19, 2022. Retrieved May 11, 2015.
- ^ Larry Wall (November 29, 1984). "patch version 1.2--YOU WANT THIS". Newsgroup: net.sources. Usenet: 1508@sdcrdcf.UUCP. Archived from the original on March 21, 2020. Retrieved May 11, 2015.
- ^ Larry Wall (May 8, 1985). "patch version 1.3". Newsgroup: net.sources. Usenet: 813@genrad.UUCP. Archived from the original on February 19, 2022. Retrieved May 11, 2015.
- ^ David MacKenzie; Paul Eggert; Richard Stallman (1997). Comparing and Merging Files with GNU Diff and Patch. Bristol: Network Theory. ISBN 978-0-9541617-5-0. Archived from the original on 2015-03-31. Retrieved 2015-03-17.
- ^ "Detailed Description of Unified Format". GNU Diffutils (version 3.7, 7 January 2018). Archived from the original on 18 January 2020. Retrieved 29 January 2020.
- ^ van Rossum, Guido. "Unified Diff Format". All Things Pythonic. Archived from the original on 2019-12-25. Retrieved 2020-01-29.
- ^ 2.2.3 Showing Which Sections Differences Are in, GNU diffutils manual
- ^ Unified Diff Format by Guido van Rossum, June 14, 2006
- ^ http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403 Archived 2013-04-29 at the Wayback Machine Section 3.206
- ^ E. Myers (1986). "An O(ND) Difference Algorithm and Its Variations". Algorithmica. 1 (2): 251–266. CiteSeerX 10.1.1.4.6927. doi:10.1007/BF01840446. S2CID 6996809.
- ^ Webb Miller; Eugene W. Myers (1985). "A File Comparison Program". Software: Practice and Experience. 15 (11): 1025–1040. CiteSeerX 10.1.1.189.70. doi:10.1002/spe.4380151102. S2CID 15489614.
- ^ Esko Ukkonen (1985). "Algorithms for Approximate String Matching". Information and Control. 64 (1–3): 100–118. doi:10.1016/S0019-9958(85)80046-2.
- ^ GNU Diff utilities Archived 2015-03-16 at the Wayback Machine. Made available by the Free Software Foundation. Free Documentation. Free source code.
- ^ Waugh, Tim (12 June 2020). "twaugh/patchutils". GitHub. Archived from the original on 1 October 2020. Retrieved 28 June 2020.
- ^ "merge (GNU RCS 5.10.0)". gnu.org. Archived from the original on 18 September 2019. Retrieved 22 January 2021.
- ^ Moolenaar, Bram. "Vim documentation: diff". vimdoc.sourceforge.net. Archived from the original on 16 February 2020. Retrieved 1 May 2020.
The easiest way to start editing in diff mode is with the "vimdiff" command. This starts Vim as usual, and additionally sets up for viewing the differences between the arguments.
vimdiff file1 file2 [file3] [file4] [...file8]
This is equivalent to:vim -d file1 file2 [file3] [file4] [...file8]
- ^ a b Brabandt, Christian (1 December 2018). "The power of diff". Vimways. Archived from the original on 2 December 2018. Retrieved 1 May 2020.
- ^ "gnu.org". www.gnu.org. Archived from the original on 2020-08-11. Retrieved 2020-09-12.
- ^ "colordiff". www.colordiff.org. Archived from the original on 2018-06-14. Retrieved 2018-06-14.
- ^ Horwitz, Susan (June 1990). "Identifying the semantic and textual differences between two versions of a program". ACM SIGPLAN Notices. 25 (6): 234–245. CiteSeerX 10.1.1.49.3377. doi:10.1145/93548.93574. Archived from the original on 2010-06-12. Retrieved 2017-11-01.
- ^ Yang, Wuu (July 1991). "Identifying syntactic differences between two programs". Software: Practice and Experience. 21 (7): 739–755. CiteSeerX 10.1.1.13.9377. doi:10.1002/spe.4380210706. S2CID 10853673.
- ^ Grass. Cdiff: A syntax directed Diff for C++ programs. Proceedings USENIX C++ Conf., pp. 181-193, 1992
- ^ Compare++, http://www.coodesoft.com/ Archived 2011-11-29 at the Wayback Machine
- ^ SmartDifferencer, http://www.semanticdesigns.com/Products/SmartDifferencer Archived 2009-10-14 at the Wayback Machine
- ^ "xaizek/zograscope". GitHub. 26 May 2020. Archived from the original on 21 December 2020. Retrieved 27 June 2020.
- ^ DaisyDiff, https://code.google.com/p/daisydiff/ Archived 2015-03-19 at the Wayback Machine
- ^ xmldiffpatch, http://msdn.microsoft.com/en-us/library/aa302294.aspx Archived 2009-10-27 at the Wayback Machine
- ^ xmldiffmerge, http://www.alphaworks.ibm.com/tech/xmldiffmerge Archived 2009-09-24 at the Wayback Machine
- ^ Cheney, Austin. Pretty Diff - Documentation. http://prettydiff.com/documentation.php Archived 2012-07-31 at the Wayback Machine
- ^ a b dontcallmedotcom. "spiff". GitHub. Archived from the original on 2015-03-26. Retrieved 2013-06-16.
- ^ a b Nachbar, Daniel W (1999-12-01). "HP-UX Porting and Archiving". UK. Archived from the original on 2012-09-05. Retrieved 2013-06-13.
- ^ "SPIFF 1". 1988-02-02. Archived from the original on 2016-10-02. Retrieved 2013-06-16.
- ^ Nachbar, Daniel W (1988-02-02). "Man page". UK. Archived from the original on 2012-09-10. Retrieved 2013-06-16.
- ^ Davide (2009-09-28). "stackoverflow". Archived from the original on 2022-02-19. Retrieved 2013-06-16.
- ^ Libenzi, Davide. "LibXDiff". SourceForge FreshMeat. Archived from the original on 2020-07-01. Retrieved 2020-06-28.
- ^ Nugroho, Yusuf Sulistyo; Hata, Hideaki; Matsumoto, Kenichi (January 2020). "How different are different diff algorithms in Git?: Use --histogram for code changes". Empirical Software Engineering: 790–823. doi:10.1007/s10664-019-09772-z. S2CID 59608676.
- ^ "algorithm - What's the difference between 'git diff --patience' and 'git diff --histogram'?". Stack Overflow. Archived from the original on 2022-02-19. Retrieved 2020-06-28.
This does indeed show that histogram diff slightly beats Myers, while patience is much slower than the others.
Further reading
- Paul Heckel (April 1978). "A technique for isolating differences between files". Communications of the ACM. 21 (4): 264–268. doi:10.1145/359460.359467. S2CID 207683976.A technique for isolating differences between files
- A generic implementation of the Myers SES/LCS algorithm with the Hirschberg linear space refinement (C source code)
External links
- The Single UNIX Specification, Version 4 from The Open Group : compare two files – Shell and Utilities Reference,
- Plan 9 Programmer's Manual, Volume 1 –
- Inferno General commands Manual –
- File comparison at Curlie
- JavaScript Implementation