1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
|
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" >
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" >
<title>Documentation - fast-cpp-csv-parser - Fast C++ CSV Parser
</title>
</head>
<div id="wikicontent">
<div class="vt" id="wikimaincol">
<h1><b>Documentation - fast-cpp-csv-parser</b></h1>
<p>The libary provides two classes: </p><ul><li><tt>LineReader</tt>: A class to efficiently read large files line by line. </li><li><tt>CSVReader</tt>: A class that efficiently reads large CSV files. </li></ul><p>Note that everything is contained in the <tt>io</tt> namespace. </p><h1><a name="LineReader"></a><tt>LineReader</tt><a href="#LineReader" class="section_anchor"></a></h1><pre class="prettyprint">class LineReader{
public:
// Constructors
LineReader(some_string_type file_name);
LineReader(some_string_type file_name, std::FILE*file);
// Reading
char*next_line();
// File Location
void set_file_line(unsigned);
unsigned get_file_line(unsigned)const;
void set_file_name(some_string_type file_name);
const char*get_truncated_file_name()const;
};</pre><p>The constructor takes a file name and optionally a <tt>stdio.h</tt> file handle. If no file handle is provided the class tries to open the file and throws an <tt>error::can_not_open_file exception</tt> on failure. If a file handle is provided then the file name is only used to format error messages. The library will call <tt>std::fclose</tt> on the file handle. <tt>some_string_type</tt> can be a <tt>std::string</tt> or a <tt>char*</tt>. </p><p>Lines are read by calling the <tt>next_line</tt> function. It returns a pointer to a null terminated C-string that contains the line. If the end of file is reached a null pointer is returned. The newline character is not included in the string. You may modify the string as long as you do not write past the null terminator. The string stays valid until the destructor is called or until next_line is called again. Windows and <tt>*</tt>nix newlines are handled transparently. UTF-8 BOMs are automatically ignored and missing newlines at the end of the file are no problem. </p><p><strong>Important:</strong> There is a limit of 2^24-1 characters per line. If this limit is exceeded a <tt>error::line_length_limit_exceeded</tt> exception is thrown. </p><p>Looping over all the lines in a file can be done in the following way. </p><pre class="prettyprint">LineReader in(...);
while(char*line = in.next_line()){
...
}</pre><p>The remaining functions are mainly used used to format error messages. The file line indicates the current position in the file, i.e., after the first <tt>next_line</tt> call it is 1 and after the second 2. Before the first call it is 0. The file name is truncated as internally C-strings are used to avoid <tt>std::bad_alloc</tt> exceptions during error reporting. </p><p><strong>Note:</strong> It is not possible to exchange the line termination character. </p><h1><a name="CSVReader"></a><tt>CSVReader</tt><a href="#CSVReader" class="section_anchor"></a></h1><p><tt>CSVReader</tt> uses policies. These are classes with only static members to allow core functionality to be exchanged in an efficient way. </p><pre class="prettyprint">template<
unsigned column_count,
class trim_policy = trim_chars<' ', '\t'>,
class quote_policy = no_quote_escape<','>,
class overflow_policy = throw_on_overflow,
class comment_policy = no_comment
>
class CSVReader{
public:
// Constructors
CSVReader(some_string_type file_name);
CSVReader(some_string_type file_name, std::FILE*file);
// Parsing Header
void read_header(ignore_column ignore_policy, some_string_type col_name1, some_string_type col_name2, ...);
void set_header(some_string_type col_name1, some_string_type col_name2, ...);
bool has_column(some_string_type col_name)const;
// Read
bool read_row(ColType1&col1, ColType2&col2, ...);
// File Location
void set_file_line(unsigned);
unsigned get_file_line(unsigned)const;
void set_file_name(some_string_type file_name);
const char*get_truncated_file_name()const;
};</pre><p>The <tt>column_count</tt> template parameter indicates how many columns you want to read from the CSV file. This must not necessarily coincide with the actual number of columns in the file. The three policies govern various aspects of the parsing. </p><p>The trim policy indicates what characters should be ignored at the begin and the end of every column. The default ignores spaces and tabs. This makes sure that </p><pre class="prettyprint">a,b,c
1,2,3</pre><p>is interpreted in the same way as </p><pre class="prettyprint"> a, b, c
1 , 2, 3</pre><p>The trim_chars can take any number of template parameters. For example <tt>trim_chars<' ', '\t', '_'> </tt>is also valid. If no character should be trimmed use <tt>trim_chars<></tt>. </p><p>The quote policy indicates how string should be escaped. It also specifies the column separator. The predefined policies are: </p><ul><li><tt>no_quote_escape<sep></tt> : Strings are not escaped. "<tt>sep</tt>" is used as column separator. </li><li><tt>double_quote_escape<sep, quote></tt> : Strings are escaped using quotes. Quotes are escaped using two consecutive quotes. "<tt>sep</tt>" is used as column separator and "<tt>quote</tt>" as quoting character. </li></ul><p><strong>Important</strong>: Quoting can be quite expensive. Disable it if you do not need it. </p><p>The overflow policy indicates what should be done if the integers in the input are too large to fit into the variables. There following policies are predefined: </p><ul><li><tt>throw_on_overflow</tt> : Throw an <tt>error::integer_overflow</tt> or <tt>error::integer_underflow</tt> exception. </li><li><tt>ignore_overflow</tt> : Do nothing and let the overflow happen. </li><li><tt>set_to_max_on_overflow</tt> : Set the value to <tt>numeric_limits<...>::max()</tt> (or to the min-pendant). </li></ul><p>The comment policy allows to skip lines based on some criteria. Valid predefined policies are: </p><ul><li><tt>no_comment</tt> : Do not ignore any line. </li><li><tt>empty_line_comment</tt> : Ignore all lines that are empty or only contains spaces and tabs. </li><li><tt>single_line_comment<com1, com2, ...></tt> : Ignore all lines that start with com1 or com2 or ... as the first character. There may not be any space between the beginning of the line and the comment character. </li><li><tt>single_and_empty_line_comment<com1, com2, ...></tt> : Ignore all empty lines and single line comments. </li></ul><p>Examples: </p><ul><li><tt>CSVReader<4, trim_chars<' '>, double_quote_escape<',','\"'> ></tt> reads 4 columns from a normal CSV file with string escaping enabled. </li><li><tt>CSVReader<3, trim_chars<' '>, no_quote_escape<'\t'>, single_line_comment<'#'> ></tt> reads 3 columns from a tab separated file with string escaping disabled. Lines starting with a # are ignored. </li></ul><p>The constructors and the file location functions are exactly the same as for <tt>LineReader</tt>. See its documentation for details. </p><p>There are three methods that deal with headers. The <tt>read_header</tt> methods reads a line from the file and rearranges the columns to match that order. It also checks whether all necessary columns are present. The <tt>set_header</tt> method does <strong>not</strong> read any input. Use it if the file does not have any header. Obviously it is impossible to rearrange columns or check for their availability when using it. The order in the file and in the program must match when using <tt>set_header</tt>. The <tt>has_column</tt> method checks whether a column is present in the file. The first argument of <tt>read_header</tt> is a bitfield that determines how the function should react to column mismatches. The default behavior is to throw an <tt>error::extra_column_in_header</tt> exception if the file contains more columns than expected and an <tt>error::missing_column_in_header</tt> when there are not enough. This behavior can be altered using the following flags. </p><ul><li><tt>ignore_no_column</tt>: The default behavior, no flags are set </li><li><tt>ignore_extra_column</tt>: If a column with a name is in the file but not in the argument list, then it is silently ignored. </li><li><tt>ignore_missing_column</tt>: If a column with a name is not in the file but is in the argument list, then <tt>read_row</tt> will not modify the corresponding variable. </li></ul><p>When using <tt>ignore_column_missing</tt> it is a good idea to initialize the variables passed to <tt>read_row</tt> with a default value, for example: </p><pre class="prettyprint">// The file only contains column "a"
CSVReader<2>in(...);
in.read_header(ignore_missing_column, "a", "b");
int a,b = 42;
while(in.read_row(a,b)){
// a contains the value from the file
// b is left unchanged by read_row, i.e., it is 42
}</pre><p>If only some columns are optional or their default value depends on other columns you have to use <tt>has_column</tt>, for example: </p><pre class="prettyprint">// The file only contains the columns "a" and "b"
CSVReader<2>in(...);
in.read_header(ignore_missing_column, "a", "b", "sum");
if(!in.has_column("a") || !in.has_column("b"))
throw my_neat_error_class();
bool has_sum = in.has_column("sum");
int a,b,sum;
while(in.read_row(a,b,sum)){
if(!has_sum)
sum = a+b;
}</pre><p><strong>Important</strong>: Do not call <tt>has_column</tt> from within the read-loop. It would work correctly but significantly slowdown processing. </p><p>If two columns have the same name an error::duplicated_column_in_header exception is thrown. If <tt>read_header</tt> is called but the file is empty a <tt>error::header_missing</tt> exception is thrown. </p><p>The <tt>read_row</tt> function reads a line, splits it into the columns and arranges them correctly. It trims the entries and unescapes them. If requested the content is interpreted as integer or as floating point. The variables passed to read_row may be of the following types. </p><ul><li>builtin signed integer: These are <tt>signed char</tt>, <tt>short</tt>, <tt>int</tt>, <tt>long</tt> and <tt>long long</tt>. The input must be encoded as a base 10 ASCII number optionally preceded by a + or -. The function detects whether the integer is too large would overflow (or underflow) and behaves as indicated by overflow_policy. </li><li>builtin unsigned integer: Just as the signed counterparts except that a leading + or - is not allowed. </li><li>builtin floating point: These are <tt>float</tt>, <tt>double</tt> and <tt>long double</tt>. The input may have a leading + or -. The number must be base 10 encoded. The decimal point may either be a dot or a comma. (Note that a comma will only work if it is not also used as column separator or the number is escaped.) A base 10 exponent may be specified using the "1e10" syntax. The "e" may be lower- or uppercase. Examples for valid floating points are "1", "-42.42" and "+123.456E789". The input is rounded to the next floating point or infinity if it is too large or small. </li><li><tt>char</tt>: The column content must be a single character. </li><li><tt>std::string</tt>: The column content is assigned to the string. The std::string is filled with the trimmed and unescaped version. </li><li><tt>char*</tt>: A pointer directly into the buffer. The string is trimmed and unescaped and null terminated. This pointer stays valid until read_row is called again or the CSVReader is destroyed. Use this for user defined types. </li></ul><p>Note that there is no inherent overhead to using <tt>char*</tt> and then interpreting it compared to using one of the parsers directly build into <tt>CSVReader</tt>. The builtin number parsers are pure convenience. If you need a slightly different syntax then use <tt>char*</tt> and do the parsing yourself. </p>
</div>
</div>
</td><tr>
</table>
</div>
</body>
</html>
|
