﻿a b c d←⍎¨('\.'⎕R'\n'),⊂2⊃⎕WG'APLVersion'
{(⍵[1]≠16):{⎕←'*** Demo requires 16.0 ***' ⋄ ⎕SIGNAL 16}0}a c
⎕NUNTIE ⎕NNUMS
1 ⎕NDELETE¨'ToBe.txt' 'ToBe2.txt' 'xml.txt' 'xml2.txt' 'xml3.txt' 'Example.csv' 'Exercise2.csv' 'Exercise2A.csv' 'Exercise2B.csv'
src←'C:\Users\richard.DYALOG\Documents\Conferences\2017 User Meeting\TextFileWorkshop\'
'ToBe.txt'⎕NPUT⍨((⊂'Windows-1252')@2)⎕NGET src,'ToBe.txt'
{⍵ ⎕NPUT⍨⎕NGET src,⍵}¨'xml.txt' 'xml2.txt' 'xml3.txt' 'Example.csv' 'Exercise2.csv'
PW←⎕PW
&!
&→TXTFILE

&⍝ Slides: Introduction (slides 0 to 2)

&⍝ ---------------------------------------------
&:TXTFILE

⍝ Slides: text file encodings (slides 3 to 12)

&⍝ Functions to read text files through various versions of Dyalog

⎕sh 'notepad ToBe.txt'

⍝
&⍝ Originally only ⎕NREAD
&⍝ ⎕NREAD can only read 8/16/32-bit chars
&⍝ This file is Windows-1252 encoded

t←'ToBe.txt' ⎕NTIE 0
x←⎕NREAD t 80 (⎕NSIZE t)
⎕NUNTIE t
]box on -style=min
x
⍝ Note that there appears to be some missing characters
⍴x
100↑⎕UCS x
⍝ Observe characters: 13, 10 and 146
]open https://en.wikipedia.org/wiki/Windows-1252

⍝
&⍝ Pre-15.0: ⎕R can be used to read text files (very inefficiently)

t←'ToBe.txt' ⎕NTIE 0
x←('.*'⎕R'\0'⍠'InEnc' 'Windows-1252')t
⎕NUNTIE t
x
⍴x
↑x

⍝
&⍝ 15.0: ⎕NGET can read and decode the file
&⍝ It does not even need to be told the encoding
&⍝ It normalises line endings

x←⎕NGET 'ToBe.txt' 1
x
⍴⊃x
↑⊃x
100↑⎕UCS⊃⎕NGET 'ToBe.txt'
⍝ Observe characters: 13, 10 and 8217
1↓⎕NGET 'ToBe.txt'

⍝ Slides: File encoding deduction (slides 13 to 44)

⍝
&⍝ To illustrate deduction, write the same text back encoded as UTF-8 without a BOM
&⍝ and read it again without specifying the encoding

'ToBe2.txt' ⎕NPUT⍨((⊂'UTF-8-NOBOM')@2)⎕NGET 'ToBe.txt'
1↓⎕NGET 'ToBe2.txt'

⍝
&⍝ Using ⎕R/⎕S to read a file is still useful - here ⎕S reads a file filtering
&⍝ out all lines except those with 'bear' on them. The other lines don't get
&⍝ brought into the workspace at all.
&⍝ (This works like the UNIX grep command. This presenter also has it on Windows.)
 
⎕CMD 'grep bear ToBe.txt'
t←'ToBe2.txt' ⎕NTIE 0
x←('bear'⎕S'%')t
⎕NUNTIE t
↑x
]box off

&⍝ ---------------------------------------------
&:XML1

⍝ Slides: XML (slides 45 to 52)

⍝
&⍝ Parse some XML and create a matrix

↑⊃⎕NGET 'xml.txt'1
]box on -style=min
⎕XML ⊃⎕NGET 'xml.txt'
↑⊃⎕NGET 'xml2.txt'1
⎕XML ⊃⎕NGET 'xml2.txt'
]box off
⍝ To be continued...

&⍝ ---------------------------------------------
&:JSON

⎕PW←9999

⍝
&⍝ In this section we are going to get the DyalogAPL posts from Facebook's servers.
&⍝ They are delivered in JSON format.

]open https://www.facebook.com/DyalogAPL/

⍝
&⍝ Presenter: show the feed (and make sure logged on)

]open https://developers.facebook.com/tools/explorer

⍝
&⍝ Presenter:
&⍝ * Get a token for User Posts
&⍝ * Explain temp/permanent token difference
&⍝ * Get: DyalogAPL?fields=feed.limit(9999)
&⍝ * Observe feed; structure of JSON; some things are messages, some are stories

&⍝ Obtain the data using Conga

]load HTTPCommand
⍝ Note: paste in the correct token!
fb←#.HttpCommand.Get 'https://graph.facebook.com/v2.10/DyalogAPL?fields=feed.limit(9999)&access_token=<TOKEN>'
100↑fb.Data

⍝ Slides: JSON elements (slides 53 to 65)

⍝
&⍝ Parse the JSON and create data

fb1←⎕JSON fb.Data
fb1.feed
⍴fb1.feed.data
fb1.feed.data[1]
⍝ Change the index numbers below if needed!
fb1.feed.data[1].message
fb1.feed.data[8].message ⍝ Will error
fb1.feed.data[8].story

⍝
&⍝ Parse the JSON and create a matrix

fb2←(⎕JSON⍠'M') fb.Data
8↑fb2

⍝
&⍝ Exercise 1
&⍝ Put the file on the internet so that it can be obtained for the exercise

⍝ Ensure connected to home VPN!
fb.Data ⎕NPUT'\\red.broadmeadow.eu\www\Dyalog\Exercise1.txt' 1
&⍎1 ⎕NDELETE 'c:\tmp\Exercise1.txt'
⍝ Save Exercise1.txt in c:\tmp
]open http://broadmeadow.eu/Dyalog/Workshop.html
data←⊃⎕NGET 'c:\tmp\Exercise1.txt'
100↑data
⍝ How many of the Facebook posts contain the word 'share'?
⍝ Care: the word appears elsewhere in the file!
⍝ Solution 1...
↑{d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v⋄''}¨↓10↑fb2
↑('share'⎕S '%'){d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v⋄''}¨↓fb2
⍴('share'⎕S '%'){d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v⋄''}¨↓fb2
↑('\bshare\b'⎕S '%'){d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v⋄''}¨↓fb2
⍴('\bshare\b'⎕S '%'){d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v⋄''}¨↓fb2
⍴('\bshare\b'⎕S '%'){d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v⋄''}¨↓(⎕JSON⍠'M') data
fn←{⍴(('\b',⍺,'\b')⎕S'%'){d n v t←⍵ ⋄ (d=4)∧(n≡'message'):v ⋄ ''}¨↓(⎕JSON⍠'M')⍵}
'share' fn data
⍝ Solution 2...
fb1.feed.data
⍴fb1.feed.data
{fb1.feed.data[⍵].message}1
{fb1.feed.data[⍵].message}¨⍳10
{6::'' ⋄ fb1.feed.data[⍵].message}¨⍳10
↑('\bshare\b'⎕S '%'){6::'' ⋄ fb1.feed.data[⍵].message}¨⍳⍴fb1.feed.data
⍴('\bshare\b'⎕S '%'){6::'' ⋄ fb1.feed.data[⍵].message}¨⍳⍴fb1.feed.data
⍝
&⍝ Useful tip

⎕JSON'{"a": 123 "b":456}'
⎕DMX
(⎕JSON⍠'Compact'0) ⎕DMX

&⍝ ---------------------------------------------
&:XML2

⎕PW←9999

⍝
&⍝ Parse some XML and create a matrix

⊃⎕NGET 'xml3.txt'
]box on -style=min
⎕XML ⊃⎕NGET 'xml3.txt'

⍝
&⍝ Illustrate how the XML can be converted to JSON, which would enable us to load as actual data
⍝ Ensure not connected to home VPN!
]open http://www.utilities-online.info/xmltojson/?save=b1330c50-53fd-402a-b2e0-4b6bea620e21-xmltojson

⍝
&⍝ This converter even tries to deal with "irregular" XML

]open http://www.utilities-online.info/xmltojson/?save=cee761b8-3321-4864-a74c-e6f2c061b57a-xmltojson

&⍝ ---------------------------------------------
&:CSV

⎕PW←9999

⍝ Slides: CSV (slides 66 to 72)

&⍝ Process the file

]open Example.csv
]box on -style=min
⍝
&⍝ Read (don't process) the CSV file
⊃⎕NGET 'Example.csv'
⍝
&⍝ Read then process the CSV file
⎕CSV(⊃⎕NGET'Example.csv') 'S'
⍝
&⍝ ⎕CSV can read the file directly
⎕CSV 'Example.csv'
⍝
&⍝ Inverted format; separated header
(⎕CSV⍠'Invert'2)'Example.csv'
(⎕CSV⍠'Invert'2)'Example.csv'  '' ⍬ 1
⍝
&⍝ Conversion of numeric data (only if requested)
x←⎕←⎕CSV'Example.csv'
x[2 3;3]
⍝ Calculate the average score
0.5×+/x[2 3;3] ⍝ Will error
⎕DR¨x
x←⎕←⎕CSV'Example.csv' '' 4
⎕DR¨x
0.5×+/x[2 3;3]
⍝
&⍝ Exercise
⊃⎕NGET 'Exercise2.csv'
]open 'Exercise2.csv'
⎕CSV 'Exercise2.csv'
⍝ Convert the file so that it contains comma separators and decimal points
⍝ Convert the file so that currency values are in pounds not euros
⍝ Solutions...
∆CSV←⎕CSV⍠('Separator' ';')('Decimal' ',')
∆CSV'Exercise2.csv'
∆CSV'Exercise2.csv' '' 4
∆CSV'Exercise2.csv' '' (2 1 2 2 2) 1
'Exercise2A.csv' ⎕CSV⍨ ∆CSV'Exercise2.csv' '' (2 1 2 2 2) 1
⊃⎕NGET 'Exercise2A.csv'
]open Exercise2A.csv
d←∆CSV'Exercise2.csv' '' (2 1 2 2 2) 1
d
⊃d[1]
(⊃d[1])[;3 4 5]
(⊃d[1])[;3 4 5]÷←1.1
d
d ⎕CSV 'Exercise2B.csv'
]open Exercise2B.csv

]box off

⍝ Slides: 73-