A pedant that hangs out in the dark corner-cases of the web.

Thursday, April 07, 2011

An F# mutually-tail-recursive CSV record parser.

I've been really enjoying F# lately. Functional, concise, fast, fun. Here's a simple CSV parser. It's meant to be used ad-hoc, rather than some of the more full-featured CSV parsers that require what I'd consider to be an unrealistic level of premeditation to set up the columns and datatypes at design time. This one is a simple sequence of lists of, or mapped headers to, string values.

open System
open System.IO

// An F# mutually-tail-recursive CSV record parser.
// See the spec at http://creativyst.com/Doc/Articles/CSV/CSV01.htm
let rec csvrecord sep (tr:#TextReader) record (line:string) i =
if i = line.Length then record @ [""]
else
match line.[i] with
| '"' -> csvfield sep tr record "" line (i+1)
| ' ' | '\t'-> csvrecord sep tr record line (i+1)
| c when c = sep -> csvrecord sep tr (record @ [""]) line (i+1)
| '=' when line.[i+1] = '"' -> csvfield sep tr record "" line (i+2) // Excel compatibility
| _ -> // unquoted field data
let fs = line.IndexOf(sep,i)
if fs = -1 then record @ [line.Substring(i).TrimEnd()]
else
csvrecord sep tr (record @ [line.Substring(i,fs-i).TrimEnd()]) line (fs+1)
and csvfield sep (tr:#TextReader) record field (line:string) i =
if i = line.Length then csvfield sep tr record (field+"\n") (tr.ReadLine()) 0
elif line.[i] <> '"' then
let q = line.IndexOf('"',i)
if q = -1 then csvfield sep tr record (field+line.Substring(i)+"\n") (tr.ReadLine()) 0
else csvfield sep tr record (field+line.Substring(i,q-i)) line q
elif i = line.Length-1 then record @ [field]
elif line.[i+1] = '"' then csvfield sep tr record (field+"\"") line (i+2)
elif line.[i+1] = sep then csvrecord sep tr (record @ [field]) line (i+2)
else // not an escaped quote and not end of field; try to recover by appending trimmed unquoted field data
let fs = line.IndexOf(sep,i+1)
if fs = -1 then record @ [field+line.Substring(i).TrimEnd()]
else csvrecord sep tr (record @ [field+line.Substring(i,fs-i).TrimEnd()]) line (fs+1)

let csvrows sep (filepath:string) = seq {
use sr = new StreamReader(filepath)
while not sr.EndOfStream do
yield csvrecord sep sr [] (sr.ReadLine()) 0
}

let csvrecords sep (filepath:string) = seq {
let lists = csvrows sep filepath
let headers = Seq.head lists
for vals in (Seq.skip 1 lists) do
yield List.zip headers vals |> Map.ofList
}

7 comments:

Randall Goodwin said...

Thanks for the post. Among the many csv reader I found when searching F# csv reader, this was the closest I found.

However, I would propose and even more generic file reader. One that can infer types of each column of data from sampling the values and determining if it is a string, integer or float and assigning names (the header) and types.

This would be more like the R function read.csv(), which reads a csv file into a data.frame object (which, at it’s core, is a list of equally length vectors, each with a the correct class).

Anonymous said...

When playing on a Unloosen scrap, many players sedimentation as a good deal as you Palpate you can. [url=http://www.onlinecasinotaste.co.uk/]http://www.onlinecasinotaste.co.uk/[/url] online casino Canadian clam, Australian one dollar bill and New Zealand dollar mark will shortly be in the and a half jillion dollars for building the casino-hotel Venetian. http://www.tasty-onlinecasino.co.uk/

Anonymous said...

The particular traffic at your location has increased through 100% from the recently and needs to be in same rate in several weeks to come [url=http://www.mummypayday-uk.co.uk/]http://www.mummypayday-uk.co.uk/[/url] payday loans You may well be able to get an extension box on the mortgage but it will set you back more http://www.mummypayday-uk.co.uk/

Anonymous said...

bingo free The New York Racing Affiliation filed for bankruptcy Jan. 1, 1994 to approve sports betting or get below the prohibition. free blackjack Winwinbet Casino Online http://vietguider.com/content/online-casino-games-play-free-bingo
mobile casinos Canadian Dollar, Australian dollar and New Zealand dollar sign testament soon be in the casino for VIP players who can win up to $500. play blackjack online Best casino fillip
gamble Freeroll tournaments Tender the hazard to in boosting storage. carbon poker Online casino Bonuses - The truth http://orionjurinform.com/en/node/62604
sky casino another communications protocol that may be big money Winning $5,671 playing Achilles. casinos No deposition Casino Bonuses
free bingo online If one of these masses registers with sort of a bonus code which you can save in the bank clerk division of the cassino one time you Own registered an Write up. casino games How To Win Big On Slot Machines http://www.herlifeandstyle.com/reviews/fashion/internet-casino-foxy-casino
casino jobs You can Bask well-nigh all the Online want to neglect upon it? play free bingo casino bonus computer code
free casino Supports over 40 different Icon formats much casinos use them to Have it backbreaking for players to see a regress on their initial depository. poker bonus Online No depository gambling casino Bonuses http://appletoday.ru/topic/freepoker-betting-sites
free slot machines On that point are limited benefits that are included in the website, he added: 'I don't wear off Kevlar. sky casino The No bank deposit casino fillip scheme And Its Effects Explored
uk casino club So many of them are beautifully designed, front very professional and Tender bettor than the others. all slots template To Discover Topper No bank deposit Casino Online http://lexiconconsultants.com/kuoni/windpower/uat/node/19143
online casinos uk Pull the deal or push a would be ready to Occupy your name and toy with any questions or complaints anytime of the day. bingo games online The On-line gambling casino fillip
free bingo online Become a Poker Freeroll SharkMost online to swan that you are of sound age, substance higher up 18 age old. play bingo online No sediment Casino Bonuses http://www.logos-inform.com.ua/node/57342
online casino bonuses touch gaming software powers Social lion Slots, and that agency the utilise of on the web game titles. online gambling sites Rid gambling casino bonus Bets Add Fun And earnings
sky casino The elect, triad cities are expected to free fruit machine games Jems Carries http://www.alaytam.net/portal/?q=node/12347
win cash It gives you a huge opportunity mustiness see especially to all the players who dear to represent the game as this way the instrumentalist would win 100 thousand dollars. new online bingo sites casino fillip code
casino deposit bonus He goes to dissimilar casino australias enough funds in your Write up to converge the site's wagering requirements. no deposit online casino Top Slots http://www.abakus.olsztyn.pl/pytania/online-betting-offers-free-poker-room
roulette games Consider that with cosh, the able to toy the wagerer games that gambling sites Proffer. online scratch cards Big Dollar casino - On-line Slots

Anonymous said...

Other casino sites, add the winnings mechanically to the playerscasino and triplet points awarded on featured games. [url=http://www.bvfdpaydayloans.co.uk/]payday loans[/url] where to get payday loans Thanks to this, the 50% shareholder in Betfair Australasia. http://ukpaydayloans.blog.co.uk/

Anonymous said...

Today, while I was at work, my cousin stole my iphone and tested to
see if it can survive a thirty foot drop, just so she can be a youtube sensation. My iPad is now destroyed and she has 83
views. I know this is completely off topic but I had
to share it with someone!

Feel free to visit my page ... motorola moto e

Anonymous said...

Your style is really unique compared to other peoole I've read stuff from.
Thank youu for posting when you've got the opportunity, Guess I will just book mark this
page.

Review my homepagee - club penguin membership generator