]>
diplodocus.org Git - nmh/blob - sbr/m_getfld.c
edbf11eab0eab2c694ea00a64ce8fdd22f269c87
3 * m_getfld.c -- read/parse a message
5 * This code is Copyright (c) 2002, by the authors of nmh. See the
6 * COPYRIGHT file in the root directory of the nmh distribution for
7 * complete copyright information.
14 /* This module has a long and checkered history. First, it didn't burst
15 maildrops correctly because it considered two CTRL-A:s in a row to be
16 an inter-message delimiter. It really is four CTRL-A:s followed by a
17 newline. Unfortunately, MMDF will convert this delimiter *inside* a
18 message to a CTRL-B followed by three CTRL-A:s and a newline. This
19 caused the old version of m_getfld() to declare eom prematurely. The
20 fix was a lot slower than
22 c == '\001' && peekc (iob) == '\001'
24 but it worked, and to increase generality, MBOX style maildrops could
25 be parsed as well. Unfortunately the speed issue finally caught up with
26 us since this routine is at the very heart of MH.
28 To speed things up considerably, the routine Eom() was made an auxilary
29 function called by the macro eom(). Unless we are bursting a maildrop,
30 the eom() macro returns FALSE saying we aren't at the end of the
33 The next thing to do is to read the mts.conf file and initialize
34 delimiter[] and delimlen accordingly...
36 After mhl was made a built-in in msh, m_getfld() worked just fine
37 (using m_unknown() at startup). Until one day: a message which was
38 the result of a bursting was shown. Then, since the burst boundaries
39 aren't CTRL-A:s, m_getfld() would blinding plunge on past the boundary.
40 Very sad. The solution: introduce m_eomsbr(). This hook gets called
41 after the end of each line (since testing for eom involves an fseek()).
42 This worked fine, until one day: a message with no body portion arrived.
45 while (eom (c = Getc (iob), iob))
48 loop caused m_getfld() to return FMTERR. So, that logic was changed to
49 check for (*eom_action) and act accordingly.
51 This worked fine, until one day: someone didn't use four CTRL:A's as
52 their delimiters. So, the bullet got bit and we read mts.h and
53 continue to struggle on. It's not that bad though, since the only time
54 the code gets executed is when inc (or msh) calls it, and both of these
55 have already called mts_init().
57 ------------------------
58 (Written by Van Jacobson for the mh6 m_getfld, January, 1986):
60 This routine was accounting for 60% of the cpu time used by most mh
61 programs. I spent a bit of time tuning and it now accounts for <10%
62 of the time used. Like any heavily tuned routine, it's a bit
63 complex and you want to be sure you understand everything that it's
64 doing before you start hacking on it. Let me try to emphasize
65 that: every line in this atrocity depends on every other line,
66 sometimes in subtle ways. You should understand it all, in detail,
67 before trying to change any part. If you do change it, test the
68 result thoroughly (I use a hand-constructed test file that exercises
69 all the ways a header name, header body, header continuation,
70 header-body separator, body line and body eom can align themselves
71 with respect to a buffer boundary). "Minor" bugs in this routine
72 result in garbaged or lost mail.
74 If you hack on this and slow it down, I, my children and my
75 children's children will curse you.
77 This routine gets used on three different types of files: normal,
78 single msg files, "packed" unix or mmdf mailboxs (when used by inc)
79 and packed, directoried bulletin board files (when used by msh).
80 The biggest impact of different file types is in "eom" testing. The
81 code has been carefully organized to test for eom at appropriate
82 times and at no other times (since the check is quite expensive).
83 I have tried to arrange things so that the eom check need only be
84 done on entry to this routine. Since an eom can only occur after a
85 newline, this is easy to manage for header fields. For the msg
86 body, we try to efficiently search the input buffer to see if
87 contains the eom delimiter. If it does, we take up to the
88 delimiter, otherwise we take everything in the buffer. (The change
89 to the body eom/copy processing produced the most noticeable
90 performance difference, particularly for "inc" and "show".)
92 There are three qualitatively different things this routine busts
93 out of a message: field names, field text and msg bodies. Field
94 names are typically short (~8 char) and the loop that extracts them
95 might terminate on a colon, newline or max width. I considered
96 using a Vax "scanc" to locate the end of the field followed by a
97 "bcopy" but the routine call overhead on a Vax is too large for this
98 to work on short names. If Berkeley ever makes "inline" part of the
99 C optimiser (so things like "scanc" turn into inline instructions) a
100 change here would be worthwhile.
102 Field text is typically 60 - 100 characters so there's (barely)
103 a win in doing a routine call to something that does a "locc"
104 followed by a "bmove". About 30% of the fields have continuations
105 (usually the 822 "received:" lines) and each continuation generates
106 another routine call. "Inline" would be a big win here, as well.
108 Messages, as of this writing, seem to come in two flavors: small
109 (~1K) and long (>2K). Most messages have 400 - 600 bytes of headers
110 so message bodies average at least a few hundred characters.
111 Assuming your system uses reasonably sized stdio buffers (1K or
112 more), this routine should be able to remove the body in large
113 (>500 byte) chunks. The makes the cost of a call to "bcopy"
114 small but there is a premium on checking for the eom in packed
115 maildrops. The eom pattern is always a simple string so we can
116 construct an efficient pattern matcher for it (e.g., a Vax "matchc"
117 instruction). Some thought went into recognizing the start of
118 an eom that has been split across two buffers.
120 This routine wants to deal with large chunks of data so, rather
121 than "getc" into a local buffer, it uses stdio's buffer. If
122 you try to use it on a non-buffered file, you'll get what you
123 deserve. This routine "knows" that struct FILEs have a _ptr
124 and a _cnt to describe the current state of the buffer and
125 it knows that _filbuf ignores the _ptr & _cnt and simply fills
126 the buffer. If stdio on your system doesn't work this way, you
127 may have to make small changes in this routine.
129 This routine also "knows" that an EOF indication on a stream is
130 "sticky" (i.e., you will keep getting EOF until you reposition the
131 stream). If your system doesn't work this way it is broken and you
132 should complain to the vendor. As a consequence of the sticky
133 EOF, this routine will never return any kind of EOF status when
134 there is data in "name" or "buf").
140 Reads an Internet message (RFC 5322), or one or more messages stored in a
141 maildrop in mbox (RFC 4155) or MMDF format, from a file stream. Each call
142 to m_getfld() reads one header field, or a portion of the body, in sequence.
146 state: message parse state
147 bufsz: maximum number of characters to load into buf
148 iob: input file stream
152 name: header field name (array of size NAMESZ=999)
153 buf: either a header field body or message body
154 bufsz: number of characters loaded into buf
155 (return value): message parse state on return from function
159 void m_unknown(FILE *iob): Determines the message delimiter string for the
160 maildrop. Called by inc, scan, and msh when reading from a maildrop file.
162 void m_eomsbr (int (*action)(int)): Sets the hook to check for end of
163 message in a maildrop. Called only by msh.
165 Those functions save state in the State variables listed below.
169 m_getfld() retains state internally between calls in some state variables.
170 These are used for detecting the end of each message when reading maildrops:
171 static unsigned char **pat_map
172 static unsigned char *fdelim
173 static unsigned char *delimend
175 static unsigned char *edelim
179 int (*eom_action)(int)
183 m_getfld() is restricted to operate on one file stream at a time
184 because of the retained state (see "State variables" above). And the
185 first call to m_getfld() on that file stream requires that the read
186 pointer be at the beginning of the file (ftell() of 0).
190 The first call to m_getfld() on a file stream is with a state of FLD.
191 Subsequent calls provide the state returned by the previous call.
192 Therefore, given the Restrictions above, the state variable could be
193 removed from the signature and just retained internally.
199 static int m_Eom (int, FILE *);
200 static unsigned char *matchc(int, char *, int, char *);
202 #define eom(c,iob) (msg_style != MS_DEFAULT && \
203 (((c) == *msg_delim && m_Eom(c,iob)) ||\
204 (eom_action && (*eom_action)(c))))
206 static unsigned char **pat_map
;
208 static int msg_style
= MS_DEFAULT
;
211 * The "full" delimiter string for a packed maildrop consists
212 * of a newline followed by the actual delimiter. E.g., the
213 * full string for a Unix maildrop would be: "\n\nFrom ".
214 * "Fdelim" points to the start of the full string and is used
215 * in the BODY case of the main routine to search the buffer for
216 * a possible eom. Msg_delim points to the first character of
217 * the actual delim. string (i.e., fdelim+1). Edelim
218 * points to the 2nd character of actual delimiter string. It
219 * is used in m_Eom because the first character of the string
220 * has been read and matched before m_Eom is called.
222 static char *msg_delim
= "";
223 static unsigned char *fdelim
;
224 static unsigned char *delimend
;
225 static int fdelimlen
;
226 static unsigned char *edelim
;
227 static int edelimlen
;
229 static int (*eom_action
)(int) = NULL
;
231 /* This replaces the old approach, with its direct access to stdio
232 * internals. It uses one fread() to load a buffer that we manage.
234 * MSG_INPUT_SIZE is the size of the buffer.
235 * MAX_DELIMITER_SIZE is the maximum size of the delimiter used to
236 * separate messages in a maildrop, such as mbox "\nFrom ".
238 * test/inc/lots_of_headers.txt depends on the particular value of
240 #define MSG_INPUT_SIZE 8192
241 #define MAX_DELIMITER_SIZE 32
242 static struct m_getfld_buffer
{
243 unsigned char msg_buf
[2 * MSG_INPUT_SIZE
+ MAX_DELIMITER_SIZE
];
244 unsigned char *readpos
;
245 unsigned char *end
; /* One past the last character read in. */
246 /* The following support tracking of the read position in the
247 input file stream so that callers can interleave m_getfld()
248 calls with ftell() and fseek(). ytes_read replaces the old
249 m_getfld() msg_count global. last_file_pos is stored when
250 leaving m_getfld()/m_unkown(), then checked on the next entry.
251 last_internal_pos is used to remember the position used
252 internally by m_getfld() (read_more(), actually). */
254 off_t total_bytes_read
;
256 off_t last_internal_pos
;
260 enter_getfld (FILE *iob
, struct m_getfld_buffer
*m
) {
261 off_t pos
= ftello (iob
);
263 /* Rely on Restriction that the first call to m_getfld (), etc.,
264 is with the read position for the file stream set to 0. */
266 /* A new file stream, so reset the buffer state. */
267 m
->readpos
= m
->end
= m
->msg_buf
;
268 m
->total_bytes_read
= 0;
269 m
->last_file_pos
= m
->last_internal_pos
= ftello (iob
);
271 /* If the current file stream position differs from the last one,
272 then caller must have called ftell(), so adjust. */
273 if (pos
!= m
->last_file_pos
) {
276 /* Opportunity for optimization here: don't reread if the
277 new position had already been read into the buffer,
278 just move m->readpos to it. */
279 fseeko (iob
, 0, SEEK_SET
);
281 num_read
= fread (m
->msg_buf
, 1, MSG_INPUT_SIZE
, iob
);
285 /* assert (ftello (iob) == pos); */
286 m
->readpos
= m
->msg_buf
+ pos
;
287 m
->end
= m
->msg_buf
+ num_read
;
288 m
->last_internal_pos
= m
->last_file_pos
= pos
;
289 m
->total_bytes_read
= pos
;
291 /* Restore the file position that we use for the input buffer. */
292 pos
= m
->last_internal_pos
;
295 fseeko (iob
, pos
, SEEK_SET
);
302 leave_getfld (struct m_getfld_buffer
*m
, FILE *iob
) {
303 /* Save the internal file position that we use for the input buffer. */
304 m
->last_internal_pos
= ftello (iob
);
306 /* Set file stream position so that callers can use ftell (). */
307 m
->total_bytes_read
+= m
->bytes_read
;
308 fseeko (iob
, m
->total_bytes_read
, SEEK_SET
);
309 m
->last_file_pos
= ftello (iob
);
313 read_more (struct m_getfld_buffer
*m
, FILE *iob
) {
314 /* Retain at least edelimlen characters that have already been
315 read so that we can back up to them in m_Eom(). */
316 ssize_t retain
= edelimlen
;
319 if (retain
< m
->end
- m
->readpos
) retain
= m
->end
- m
->readpos
;
320 /* assert (retain <= m->readpos - m->msg_buf <= sizeof msg_buf); */
322 /* Move what we want to retain at end of the buffer to the beginning. */
323 memmove (m
->msg_buf
, m
->readpos
- retain
, retain
);
325 m
->readpos
= m
->msg_buf
+ retain
;
326 num_read
= fread (m
->readpos
, 1, MSG_INPUT_SIZE
, iob
);
327 m
->end
= m
->readpos
+ num_read
;
334 if (m
.end
- m
.readpos
< 1) {
335 if (read_more (&m
, iob
) == 0) {
336 /* Pretend that we read a character. That's what stdio does. */
343 return m
.readpos
< m
.end
? *m
.readpos
++ : EOF
;
348 int next_char
= Getc (iob
);
356 Ungetc (int c
, FILE *iob
) {
359 if (m
.readpos
== m
.msg_buf
) {
363 return *--m
.readpos
= c
;
369 m_getfld (int state
, unsigned char name
[NAMESZ
], unsigned char *buf
,
370 int *bufsz
, FILE *iob
)
372 register unsigned char *cp
;
373 register int cnt
, c
, i
, j
;
375 enter_getfld (iob
, &m
);
377 if ((c
= Getc(iob
)) < 0) {
379 leave_getfld (&m
, iob
);
384 /* flush null messages */
385 while ((c
= Getc(iob
)) >= 0 && eom (c
, iob
))
392 leave_getfld (&m
, iob
);
400 if (c
== '\n' || c
== '-') {
401 /* we hit the header/body separator */
402 while (c
!= '\n' && (c
= Getc(iob
)) >= 0) continue;
404 if (c
< 0 || (c
= Getc(iob
)) < 0 || eom (c
, iob
)) {
406 /* flush null messages */
407 while ((c
= Getc(iob
)) >= 0 && eom (c
, iob
))
413 leave_getfld (&m
, iob
);
420 * get the name of this component. take characters up
421 * to a ':', a newline or NAMESZ-1 characters, whichever
426 /* Get the field name. The first time through the loop,
427 this copies out the first character, which was loaded
428 into c prior to loop entry. Initialize j to 1 to
431 c
!= ':' && c
!= '\n' && c
!= EOF
&& j
< i
;
432 ++j
, c
= Getc (iob
)) {
436 /* Skip next character, which is either the space after
437 the ':' or the first folded whitespace. */
440 if (c
== EOF
|| (next_char
= Peek (iob
)) == EOF
) {
441 *bufsz
= *cp
= *buf
= 0;
442 advise (NULL
, "eof encountered in field \"%s\"", name
);
443 leave_getfld (&m
, iob
);
448 /* If c isn't ':' here, something went wrong. Possibilities are:
449 * . hit a newline (error)
450 * . got more than namesz chars. (error)
453 /* Finished header name, fall through to FLDPLUS below. */
454 } else if (c
== '\n') {
455 /* We hit the end of the line without seeing ':' to
456 * terminate the field name. This is usually (always?)
457 * spam. But, blowing up is lame, especially when
458 * scan(1)ing a folder with such messages. Pretend such
459 * lines are the first of the body (at least mutt also
460 * handles it this way). */
462 /* See if buf can hold this line, since we were assuming
463 * we had a buffer of NAMESZ, not bufsz. */
464 /* + 1 for the newline */
465 if (*bufsz
< j
+ 1) {
466 /* No, it can't. Oh well, guess we'll blow up. */
467 *bufsz
= *cp
= *buf
= 0;
468 advise (NULL
, "eol encountered in field \"%s\"", name
);
472 memcpy (buf
, name
, j
- 1);
475 /* The last character read was '\n'. m.bytes_read
476 (and j) include that, but it was not put into the
477 name array in the for loop above. So subtract 1. */
478 *bufsz
= --m
.bytes_read
; /* == j - 1 */
479 leave_getfld (&m
, iob
);
482 /* By design, the loop above discards the last character
483 it had read. It's in c, use it. */
485 *bufsz
= *cp
= *buf
= 0;
486 advise (NULL
, "field name \"%s\" exceeds %d bytes", name
,
492 /* Trim any trailing spaces from the end of name. */
493 while (isspace (*--cp
) && cp
>= name
) continue;
495 /* readpos points to the first character of the field body. */
500 * get (more of) the text of a field. Take
501 * characters up to the end of this field (newline
502 * followed by non-blank) or bufsz-1 characters.
509 for (finished
= 0; ! finished
; ) {
510 while (c
!= '\n' && c
!= EOF
&& j
++ < i
) {
511 *cp
++ = c
= Getc (iob
);
514 if (c
!= EOF
) c
= Peek (iob
);
516 /* the dest buffer is full */
519 } else if (c
!= ' ' && c
!= '\t') {
520 /* The next character is not folded whitespace, so
521 prepare to move on to the next field. It's OK
522 if c is EOF, it will be handled on the next
523 call to m_getfld (). */
527 /* Folded header field, continues on the next line. */
530 *bufsz
= m
.bytes_read
;
537 * get the message body up to bufsz characters or the
538 * end of the message.
543 /* Back up and store the current position and update cnt. */
545 cnt
= m
.end
- m
.readpos
;
546 c
= cnt
< i
? cnt
: i
;
547 if (msg_style
!= MS_DEFAULT
&& c
> 1) {
549 * packed maildrop - only take up to the (possible)
550 * start of the next message. This "matchc" should
551 * probably be a Boyer-Moore matcher for non-vaxen,
552 * particularly since we have the alignment table
553 * all built for the end-of-buffer test (next).
554 * But our vax timings indicate that the "matchc"
555 * instruction is 50% faster than a carefully coded
556 * B.M. matcher for most strings. (So much for elegant
557 * algorithms vs. brute force.) Since I (currently)
558 * run MH on a vax, we use the matchc instruction. --vj
562 if ((ep
= matchc( fdelimlen
, fdelim
, c
, bp
)))
566 * There's no delim in the buffer but there may be
567 * a partial one at the end. If so, we want to leave
568 * it so the "eom" check on the next call picks it up.
569 * Use a modified Boyer-Moore matcher to make this
570 * check relatively cheap. The first "if" figures
571 * out what position in the pattern matches the last
572 * character in the buffer. The inner "while" matches
573 * the pattern against the buffer, backwards starting
574 * at that position. Note that unless the buffer
575 * ends with one of the characters in the pattern
576 * (excluding the first and last), we do only one test.
581 if ((sp
= pat_map
[*ep
])) {
583 /* This if() is true unless (a) the buffer is too
584 * small to contain this delimiter prefix, or
585 * (b) it contains exactly enough chars for the
587 * For case (a) obviously we aren't going to match.
588 * For case (b), if the buffer really contained exactly
589 * a delim prefix, then the m_eom call at entry
590 * should have found it. Thus it's not a delim
591 * and we know we won't get a match.
593 if (((sp
- fdelim
) + 2) <= c
) {
595 /* Unfortunately although fdelim has a preceding NUL
596 * we can't use this as a sentinel in case the buffer
597 * contains a NUL in exactly the wrong place (this
598 * would cause us to run off the front of fdelim).
600 while (*--ep
== *--cp
)
604 /* we matched the entire delim prefix,
605 * so only take the buffer up to there.
606 * we know ep >= bp -- check above prevents underrun
612 /* try matching one less char of delim string */
614 } while (--sp
> fdelim
);
618 memcpy( buf
, bp
, c
);
619 /* Advance the current position to reflect the copy out.
620 c is less than or equal to the number of bytes remaining
621 in the read buffer, so will not overrun it. */
624 /* Subtract 1 from c because the first character was read by
625 Getc(), and therefore already accounted for in m.bytes_read. */
626 m
.bytes_read
+= c
- 1;
627 *bufsz
= m
.bytes_read
;
632 adios (NULL
, "m_getfld() called with bogus state of %d", state
);
636 leave_getfld (&m
, iob
);
646 char text
[MAX_DELIMITER_SIZE
];
648 register char *delimstr
;
650 enter_getfld (iob
, &m
);
653 * Figure out what the message delimitter string is for this
654 * maildrop. (This used to be part of m_Eom but I didn't like
655 * the idea of an "if" statement that could only succeed on the
656 * first call to m_Eom getting executed on each call, i.e., at
657 * every newline in the message).
659 * If the first line of the maildrop is a Unix "From " line, we
660 * say the style is MBOX and eat the rest of the line. Otherwise
661 * we say the style is MMDF and look for the delimiter string
662 * specified when nmh was built (or from the mts.conf file).
665 msg_style
= MS_UNKNOWN
;
667 for (c
= 0, cp
= text
; c
< 5; ++c
, ++cp
) {
668 if ((*cp
= Getc (iob
)) == EOF
) {
673 if (c
== 5 && strncmp (text
, "From ", 5) == 0) {
675 delimstr
= "\nFrom ";
676 while ((c
= Getc (iob
)) != '\n' && c
>= 0) continue;
677 /* m_unknown is only called on maildrop files, and they are only
678 read using m_getfld (). The caller musn't try to read from
679 the stream directly because the file position indicator was
680 not advanced based on bytes_read, but instead on whatever
681 was read into the message buffer. */
683 /* not a Unix style maildrop */
684 m
.readpos
-= m
.bytes_read
;
688 c
= strlen (delimstr
);
689 fdelim
= (unsigned char *) mh_xmalloc((size_t) (c
+ 3));
692 msg_delim
= (char *)fdelim
+1;
693 edelim
= (unsigned char *)msg_delim
+1;
695 edelimlen
= c
- 1; /* == strlen (delimstr) */
696 strcpy (msg_delim
, delimstr
);
697 delimend
= (unsigned char *)msg_delim
+ edelimlen
;
699 adios (NULL
, "maildrop delimiter must be at least 2 bytes");
701 * build a Boyer-Moore end-position map for the matcher in m_getfld.
702 * N.B. - we don't match just the first char (since it's the newline
703 * separator) or the last char (since the matchc would have found it
704 * if it was a real delim).
706 pat_map
= (unsigned char **) calloc (256, sizeof(unsigned char *));
708 for (cp
= (char *) fdelim
+ 1; cp
< (char *) delimend
; cp
++ )
709 pat_map
[(unsigned char)*cp
] = (unsigned char *) cp
;
711 if (msg_style
== MS_MMDF
) {
712 /* flush extra msg hdrs */
713 while ((c
= Getc(iob
)) >= 0 && eom (c
, iob
))
719 leave_getfld (&m
, iob
);
724 m_eomsbr (int (*action
)(int))
726 if ((eom_action
= action
)) {
733 msg_delim
= (char *)fdelim
+ 1;
734 fdelimlen
= strlen((char *)fdelim
);
735 delimend
= (unsigned char *)(msg_delim
+ edelimlen
);
741 * test for msg delimiter string
745 m_Eom (int c
, FILE *iob
)
748 char text
[MAX_DELIMITER_SIZE
];
751 for (i
= 0, cp
= text
; i
< edelimlen
; ++i
, ++cp
) {
752 if ((*cp
= Getc (iob
)) == EOF
) {
757 if (i
!= edelimlen
|| strncmp (text
, (char *)edelim
, edelimlen
)) {
758 if (i
== 0 && msg_style
== MS_MBOX
)
759 /* the final newline in the (brain damaged) unix-format
760 * maildrop is part of the delimitter - delete it.
764 /* Did not find delimiter, so restore the read position.
765 Note that on input, a character had already been read
766 with Getc(). It will be unget by m_getfld () on return. */
767 m
.readpos
-= m
.bytes_read
- 1;
771 if (msg_style
== MS_MBOX
) {
772 while ((c
= Getc (iob
)) != '\n')
781 static unsigned char *
782 matchc(int patln
, char *pat
, int strln
, char *str
)
784 register char *es
= str
+ strln
- patln
;
787 register char *ep
= pat
+ patln
;
788 register char pc
= *pat
++;
797 while (pp
< ep
&& *sp
++ == *pp
)
800 return ((unsigned char *)--str
);