Programming Prep

As I am considering what is next for me, a friend has been lobbying for me to interview at google. She sent me a bunch of of programming interview books and told me to talk to coach!

Coaching is something that is fascinating to me. As a high school athlete I never had a good coach, so it was never something I considered. But when I was an adult and I read a bunch of ‘life hacking’ type books, they all extolled utility of a coach. And as an adult I see the benefit, a coach is helpful to learn new skills, to move to the next level in any endeavor, and to pinpoint and work on trouble spots.

When I spoke to this programming coach, she said 75% of her clients were programmers and only 25% were job seekers. This is fascinating. What sorts of issues were they going through?  Some of the issues were regular job issues, that perhaps would have been discussed with a therapist in the past, like how do I promote my work, or talk about a problem, or deal with a difficult colleague. Other issues are how do I deal with a thorny technical problem or learn something new.

We chatted for a bit and she said some interesting things. First, she said don’t do interview prep by doing hundreds of leetcode problems. That is more important to go deep and truly understand a few problems than superficially do hundreds of problems. This actually adds to anxiety, she said. And anxiety is worse than the actual problems themselves. Instead she suggested I get a programming problems book in the language I want to focus on, make a spreadsheet of the hardest 2 or 3 problems in each chapter and every day work for 90 min or on the next problem on the list. Dont spend more than 30 min on a problem. Play with the problem, dont just start coding. Color code the problems. The ones that give you problems make red, and as they become easier make them green. When you can do the problems in your sleep you are ready.

I am really excited to try this. This morning I created my spreadsheet and I am ready to rock and roll.

Kickstarter Creative Residency

Jesse and I, as part of print all over me are doing a creative residency at Kickstarter. I love kickstarter. I remember years ago when kickstarter was kickstartr and I was like – why didn’t I think of that! Such a great solution to a problem. Print all over me in part started because people were having problems fulfilling kickstarted manufacturing projects.  I love the vibe at kickstarter. The space is beautiful, they have a garden, a library, free kombucha! The people that work there all seem fascinating individuals. Kickstarter has built such an amazing culture – it really something to admire.

At Kickstarter, ostensibly we will be working on a mental health card game based on DBT. DBT is a treatment for borderline personality disorder and cannot be entirely treaded with psychotropic drugs. It is about the dialectic between being who you are and changing.The tag line in something like: You are perfect just the way you are, just change.

However as we are doing this part of me wants to kickstart a line of code metal tee shirts, or gender fluid jumpsuits.

One thing I am going to focus on is conscious computation and a lot of the work I started at Recurse.  At the end of my time there, I am considering kickstarting the prayer blockchain glowing orb!  I am also thinking about the ideas around conscious computation, and some of the ideas around anthropology  – like the idea of prayer tech and perhaps turning that into something, perhaps a podcast, series of classes, or website.

The other residents are fascinating, one person is working on erotica, another on sci fi comics, another on African American farmers,  I could go on. I just cannot wait to talk with the other folks and learn more about what they are doing.

 

Middle – the tidalcycles python mashup

So I redid the text to voice + poem junk generation + tidalcycles. I need to save supercollider to buffer so you can’t hear my keystrokes. BUT all the sound is made with the text to speech poem generator. Thats cool.

I feel like it starts to pick up at min 1 and you can hear me curse at like  min 3.

But what happened before….

I took a look to see why my text to voice clips in tidalcycles did not work as expected . After some googling I found this great documentation.  To Quote:

As you can hear, Tidal will keep triggering the sample each cycle, even if it’s very long. Even if you stop the pattern playing, you will still need to listen while the samples play out.

This is sort of what I thought I was hearing. So how do we fix that?

something like:

d1 $ sound “bev” # cut 1

repeats the same clip again and again. Also “hush” does not work as expected because it I think needs to wait for all the cycles to finish.

Something like

d1 $ slow BIGNUM $ sound “bev ~” # cut 1

Keeps the clip going throughout many cycles

and

d1 $ chop 32 $ sound "bev"
d1 $ striate 32 $ sound "bev"

Let you play a clip from the middle!

When I went back and listened to all the clips I realize the audio did not record! Riley at RC was correct.

Here are the original clips of the two poems:

I am sticking my config here so I remember:

Quarks.checkForUpdates(); Quarks.install("SuperDirt", "v1.0")

(
s.waitForBoot {
	~dirt = SuperDirt(2, s); // two output channels

        // load samples from multiple folders:
	~dirtloadSoundFiles("/Users/Parmenides/projects/RC/tsamples/
	~dirt.loadSoundFiles("/Users/Parmenides/projects/sounds/*"); 

	s.sync; // wait for supercollider to finish booting up
	~dirt.start(57120, [0, 0]); // start superdirt, listening on port 57120, create two busses each sending audio to channel 0
};
);
SuperDirt.start

Random Lexical Experiments continued -Phonemes and Graphemes

Poem of the day: It sort of reminds me of the futuristic Hawaiian dialect from cloud atlas.

freckled te mortality
terrible violent wounded great ti take
great mighty veritable
sperm sperm commentator
sperm Norwegian ta wondrous American to certain
American sperm State
dying to Greenland to mistake
Greenland Greenland ke stand
Greenland sperm important
sperm te entire Trumpa Physeter pa momentary
Sperm Greenland ki take
right po sperm pe right right ti distance
sperm humpbacked important
Greenland Greenland captain
Greenland sperm totally
sperm Hyena instant
Tusked involuntarily
Horned te Unicorn Folio ta stand
white white white to white metaphysical
white white te stand
white white ti white Albino po sperm sperm Captain
entire sperm pe sperm white established
white sperm po tallest
sperm sperm po sperm te sperm to instance
right sperm metaphysical
English snowy entailed
true take
Right to stranded living to take
Greenland boiling foremost particular Patagonian
sperm spermaceti sperm stricken ke sperm waning te talk
same heaving understanding
tremendous great beheaded involuntarily
mightiest pa great sperm dead towing tapered
fagged pe sperm tapping
right sperm tar
sperm stricken substantial
wounded potatoes
sunken to sunken first po stricken flying stature
towing whole mountaineers
sperm towing to tablecloth
unborn stricken schoolmaster controverted te unaccountable
drugged other understand
blasted ki other lighter te Dutch slack stricken stayed
sperm white other table
adult last hunted great eternal standing
living last pa dead dead stains
stricken white famous ti white tanning
gliding sperm ki take
white white uncomfortableness
white fatal
white hated before Stammering

What I did was generate a bunch of words via regex in Moby Dick, then find pair of letter frequencies and randomly inject them, and then find word pairs within words and add them to the end of lines. I would like to experiment more with stanzas here and punctuation.

I was chatting with Colin about my python poetic experiments and he said that had built a library a long time ago for rhyming based on espeak (the cool unix version of say).  So what is a phoneme? It is a unit of sound like p or th. When you want to rhyme it is useful to know the phonemes the of word endings.  There is also the  grapheme, that is a way of writing down a phoneme. I was thinking of sound when I generated my poem today. Chapter 3 is about tokenization and fileio and some encoding so it was somewhat useful in this endeavor.

I was really interested in finding frequencies of pairs of words. Here the frequency of ka is 40, of pa 1694 (in I think Moby Dick)).

a e i o u
k 40 2727 930 24 39
p 1694 3122 1124 2372 552
r 3347 11627 4113 4672 863
s 2059 6577 2721 3199 1799
t 3339 6929 4933 7126 1669
v 737 5695 1437 481 18
>>>

Things get interesting when you load in a corpus like:

from nltk.corpus import gutenberg, nps_chat
>>> moby = nltk.Text(gutenberg.words(‘melville-moby_dick.txt’))

So if I want to match a _ whale:

>> moby.findall(r”<a> (<.*>) <whale>”)
dead; great; mightier; right; live; good; southern; white; white;
white; particular; sperm; sperm; sperm; sperm; flying; dead;
Greenland; Polar; sperm; small; dead; right; sperm; DEAD; nursing;
dead; lone; fine; blasted; second; blasted; sick; certain; discovery

or

>>> moby.findall(r”(<.*>) <fast>”)
was; it; they; a; those; him; got; when; got; locked; go; as; still;
iron; was; ;; and; be; making; so; get; him; him; when; party;
technically; technically; her; very; ,; as; walks; himself; my; the;
very; reefed; was; and; ,; held; themselves; hold; now; all; so;
travels

>> moby.findall(r”(<.*>) <blubber>”)
their; and; thousand; “; de; de; of; the; great; of; the; the; the;
the; the; his; That; same; the; or; his; the; the; fresh; the; of;
the; of; veteran; of; shrivelled; of; and; curved

This is fun! Unfortunately this prints to sysout and not to a string, so I need to do some massaging in order to make this actually usable.. BUT

Stemmers is also introduced in ch3. This is how we determine the root of a word, like “go” for “going” there are different type of stemmers and you just have to use the one you like. That is sort of the advice that the documentation gives.  The WordNetLemmanizer returns the word if it is in its dictionary. So for example if go and not going is in the dictionary go is returned.  We looked at lemmas the other day, but lemmas, similar to stemming, remove inflections/endings to return the root word.

I am not sure how this would be interesting poetically.  Perhaps you want to create a rhythm, or use root words at different parts in a line – the endings perhaps.  For languages where words are not divided by spaces, tokenization is more difficult. The analysis of segmentation addresses this and is fascinating. Basically word endings are demarcated by binary strings (1 being end of a word).  One idea is remove spaces from sections of moby dick and reconstitute them based on distribution of word length and sentence length.

Sumana posted this fantastic python library, olipy,  for this sort of poetic generation.  Oulipo is a writing style where you introduce certain constraints into the writing, like writing without the letter e. Christian Bok is one of my favorite oulipo-esque writers.  Look what happens when I google for an article on Bok and Oulipo… an article on Bok Oulipo and Bergvall come up– that my friends is synchronicity – maybe.

 

Small Experiments Continued

I was not sure what was inspiring to me today. I considered looking at some concrete poetry and a nice collection of ee cummings, but I decided to just go and click around ubuweb.  Here I became happily sidetracked with the films of Agnes Varda – who I love and listened to some Kathy Acker (trigger warnings) it hard to take.  I love the accents of midcentury native new yorkers. I hope I have something like that. A ghost of it perhaps. I finished it all off with some Samuel Beckett as a palate cleanser while coding.

Today I experimented with lexical relationships and text generation. I started with some simple definitions and examples of usage based on the word ‘middle’:  Here is the audio text to speech which is slightly more interesting. I discuss why I did this below.

Middle
it is in the center of town they ran forward into the heart of the struggle they were in the eye of the storm
a n a r e a t h a t i s a p p r o x i m a t e l y c e n t r a l w i t h i n s o m e l a r g e r r e g i o n
he hit the ball to deep center
t h e p i e c e o f g r o u n d i n t h e o u t f i e l d d i r e c t l y a h e a d o f t h e c a t c h e r
they were raising money to build a new center for research
a b u i l d i n g d e d i c a t e d t o a p a r t i c u l a r a c t i v i t y
a l o w – l y i n g r e g i o n i n c e n t r a l F r a n c e
it is in the center of town they ran forward into the heart of the struggle they were in the eye of the storm
a n a r e a t h a t i s a p p r o x i m a t e l y c e n t r a l w i t h i n s o m e l a r g e r r e g i o n

a p o i n t e q u i d i s t a n t f r o m t h e e n d s o f a l i n e o r t h e e x t r e m i t i e s o f a f i g u r e

it is in the center of town they ran forward into the heart of the struggle they were in the eye of the storm
a n a r e a t h a t i s a p p r o x i m a t e l y c e n t r a l w i t h i n s o m e l a r g e r r e g i o n
A whole is that which has beginning, middle, and end”- Aristotle
a n i n t e r m e d i a t e p a r t o r s e c t i o n
young American women believe that a bare midriff is fashionable
t h e m i d d l e a r e a o f t h e h u m a n t o r s o ( u s u a l l y i n f r o n t )

in your heart you know it is true her story would melt your bosom
t h e l o c u s o f f e e l i n g s a n d i n t u i t i o n s
he stood still, his heart thumping wildly
t h e h o l l o w m u s c u l a r o r g a n l o c a t e d b e h i n d t h e s t e r n u m a n d b e t w e e n t h e l u n g s ; i t s r h y t h m i c c o n t r a c t i o n s m o v e t h e b l o o d t h r o u g h t h e b o d y
he kept fighting on pure spunk you haven’t got the heart for baseball
t h e c o u r a g e t o c a r r y o n
t h e o r g a n o f s i g h t
she has an eye for fresh talent he has an artist’s eye
g o o d d i s c e r n m e n t ( e i t h e r v i s u a l l y o r a s i f v i s u a l l y )
he tried to catch her eye
a t t e n t i o n t o w h a t i s s e e n

The functions that generate these sentences return lists. As you can tell some are lists of words and some are lists of letters. I sort of visually like the way it looks when I concatenate indiscriminately however it is completely unreadable.

Some of these lines I love and it is sort of like an exquisite_corpse surrealist experiment.  How can I get beyond the surreal, into something new? Something with a deeper structure that gives rise to the meaning of the thing.

This is what happens when I start to play with parts of speech such as antonyms.

it is in the center of town they ran forward into the heart of the struggle they were in the eye of the storm
an area that is approximately central within some larger region

the central area on a theater stage

the central part of a city

the part of a city where financial institutions are centered
the playground is the hub of parental supervision the airport is the economic hub of the area
a center of activity or interest or commerce or transportation; a focal point around which events revolve

the older and more populated and (usually) poorer central section of a city

the part of a city where medical facilities are centered

(sports) the middle part of a playing field (as in football or lacrosse)

the middle of a stream

a center of authority (as a city from which authority is exercised)

the central area or place of lowest barometric pressure within a storm

he hit the ball to deep center
the piece of ground in the outfield directly ahead of the catcher

they were raising money to build a new center for research
a building dedicated to a particular activity

a center where patients with severe burns can be treated

a center equipped to handle a large volume of telephone calls (especially for taking orders or serving customers)

a center where the members of a community can gather for social or cultural activities

a center where conferences can be conducted
the general in command never left the control center
the operational center for a group of related activities

a center where research is done

a recreational center for servicemen

a center in an underprivileged area that provides community services

a center for student activities at a college or university

a low-lying region in central France

it is in the center of town they ran forward into the heart of the struggle they were in the eye of the storm
an area that is approximately central within some larger region

the central area on a theater stage

the central part of a city

the part of a city where financial institutions are centered
the playground is the hub of parental supervision the airport is the economic hub of the area
a center of activity or interest or commerce or transportation; a focal point around which events revolve

the older and more populated and (usually) poorer central section of a city

the part of a city where medical facilities are centered

(sports) the middle part of a playing field (as in football or lacrosse)

the middle of a stream

a center of authority (as a city from which authority is exercised)

the central area or place of lowest barometric pressure within a storm
a point equidistant from the ends of a line or the extremities of a figure

the middle part of a slack rope (as distinguished from its ends)

the center of the Earth

the center of a target

the center of the circle of curvature

the point within something at which gravity can be considered to act; in uniform gravity it is equal to the center of mass

point representing the mean position of the matter in a body
the ball has a titanium core
the center of an object

the central part of the Earth
the Incas believed that Cuzco was the navel of the universe
the center point or middle of something

the center point on a shield

the positively charged dense center of an atom

(astronomy) the center of the head of a comet; consists of small solid particles of ice and frozen gas that vaporizes on approaching the sun to form the coma and tail
it is in the center of town they ran forward into the heart of the struggle they were in the eye of the storm
an area that is approximately central within some larger region

the central area on a theater stage

the central part of a city

the part of a city where financial institutions are centered
the playground is the hub of parental supervision the airport is the economic hub of the area
a center of activity or interest or commerce or transportation; a focal point around which events revolve

the older and more populated and (usually) poorer central section of a city

the part of a city where medical facilities are centered

(sports) the middle part of a playing field (as in football or lacrosse)

the middle of a stream

a center of authority (as a city from which authority is exercised)

the central area or place of lowest barometric pressure within a storm

A whole is that which has beginning, middle, and end”- Aristotle
an intermediate part or section

young American women believe that a bare midriff is fashionable
the middle area of the human torso (usually in front)
in your heart you know it is true her story would melt your bosom
the locus of feelings and intuitions

he stood still, his heart thumping wildly
the hollow muscular organ located behind the sternum and between the lungs; its rhythmic contractions move the blood through the body

enlarged heart commonly found among athletes trained for endurance

a heart (as of mammals and birds and reptiles) having two auricles

the muscle tissue of the heart; adapted to continued rhythmic contraction

the artery that branches from the aorta to supply blood to the heart

a valve to control one-way flow of blood

a structure in a hollow organ (like the heart) with a flap to insure one-way flow of fluid through it

he kept fighting on pure spunk you haven’t got the heart for baseball
the courage to carry on

the organ of sight

in insects and some crustaceans: composed of many light-sensitive elements each forming a portion of an image
it is not safe to look directly at the sun with the naked eye
the eye unaided by any optical instrument that alters the power of vision or alters the apparent size or distance of objects

the right eye

the left eye

an informal term referring to the eye

an eye having a single lens

a natural opening in something

either of the corners of the eye where the upper and lower eyelids meet

a branch of the ophthalmic artery; enters the eyeball with the optic nerve

a highly vascular membrane in the eye between the retina and the sclera; a dark pigmentation minimizes the scattering of light inside the eye

one of several arteries supplying the choroid coat of the eye
the ciliary body produces aqueous humor
the part of the tunic of the eye between the choroid coat and the iris

a transparent lubricating mucous membrane that covers the eyeball and the under surface of the eyelid

the transparent dome-shaped anterior portion of the outer covering of the eye; it covers the iris and pupil and is continuous with the sclera

a vertical fold of skin over the nasal canthus; normal for Mongolian peoples; sometimes occurs in Down’s syndrome

the ball-shaped capsule containing the vertebrate eye
his lids would stay open no longer
either of two folds of skin that can be moved to cover or open the eye

muscular diaphragm that controls the size of the pupil which in turn controls the amount of light that enters the eye; it forms the colored portion of the eye

the structures that secrete and drain tears from the eye

an artery that originates from the ophthalmic artery and supplies the lacrimal gland and rectal eye muscles and the upper eyelid and the forehead

drains the lacrimal gland; empties into the superior ophthalmic vein

biconvex transparent body situated behind the iris in the eye; its role (along with the cornea) is to focuses light on the retina

a protective fold of skin in the eyes of reptiles and birds and some mammals

one of the small muscles of the eye that serve to rotate the eyeball

a ring of smooth muscle surrounding the iris

the innermost light-sensitive membrane covering the back wall of the eyeball; it is continuous with the optic nerve

the whitish fibrous membrane (albuginea) that with the cornea forms the outer covering and protection of the eyeball

the part of the eye that contains the iris and ciliary body and choroid

a tubule that drains excess aqueous humor

she has an eye for fresh talent he has an artist’s eye
good discernment (either visually or as if visually)

he tried to catch her eye
attention to what is seen

I feel like perhaps this is a way to generate raw material for a piece rather than the piece itself.

So I went and decided to some how use this for my daily tidal cycle/music experiment. I am using tidalcycles (vim tidal) and supercollider.  To use tidalcycles with vim (or neovim) you need to use tmux- which I do not have much experience with.

Basically tmux lets you tile windows in the command line.  At first I thought that vi was opening a bash prompt but no, the terminal was itself opening up a tidal cycles or a haskell repl.  This was confusing because I kept trying to change vi buffers and it was not working.

Ok so tmux commands:

  1.  exit – this closes application windows (detach also detaches the processes)
  2. : you can type ‘:’ to get to the tmux prompt – very VI like.
  3. ctrl-b arrow should move me to the other pane
    1. This did not work, instead i used ctrl-b o to cycle through panes

then I put SuperDirt.start (shift + return) but  error ERROR: Input sample rate is 16000, but output is 44100. Mismatched sample rates are not supported. To disable input, set the number of input channels to 0. I fixed this by opening /Applications/Utilities – Audio Midi Setup  and fixing the input rate. I had some old midi keyboard hooked up it seemed like.

A bunch of samples could not be found – oops – had to install

include("SuperDirt");

For some reason I could not get audio coming through on my earbuds – I need to debug this. I also dont have the Super Dirt samples. No matter. I played around with a bunch of random stuff and had a good time. I feel like there is no time signature there is just the cycle that we can speed up or slow down. We can put as much stuff in the cycle as we want and we can alter it programatically.   I set up a simple set of instructions and it was interesting to hear it change over time. Like a really crapy audio version of game of life.

I converted one of the poem outputs to text via Say  < poemX.txt although this seemed to loop.  When I listened to the poem it was more interesting than reading it. I noticed that middle had been interpreted as center – as in the hub of activity. This is different from the middle – as I was in the middle of my life – as in the beginning of the Divine Comedy.  I used ffmpeg to convert m4a to wav.

Basically I think the clips are too long – but these are some early experiments. I use the audio that I generated from the mac os text to speech but you cannot really tell.  Baby steps.

bell hooks and python NLP

I have a pile of to read books in the shelves by the door. Before I left the apartment for a weekend away, I grabbed a random book and it happened to be all about love by bell hooks.  It fit perfectly in my jacket pocket and I started to read it in the car.  Immediately the work resonated with me. The feeling of being lovable as a young child and then, suddenly, of being unlovable. Of spending you life looking for this love you once had, or to be the child that was once lovable, and not finding it.  Then realizing that it is futile to go back or to recover this love, since it is lost forever.  Instead, you must find a new love or new way to be loved. I am not sure if this is even what bell hooks talks about or meant, but it is what I felt.

The book is non fiction but it felt like poetry.  So I figured I would write some electronic poetry. I know this is sort of a non-sequitur when here I was talking about bell hooks. But I wondered, what makes this non-fiction instead of poetry? Is it arguments, is it language, is it marketing?  What would it look like to take a work of non-fiction, or of prose even and turn it into poetry?

One way to do this would be to train a model on a particular body of poetry and then extract the language from a non-fiction or prose book and use this lexicon to generate poetry according to the ml model. Another way to do this would be to extract each sentence and then rewrite them as lines according to the ml model. Paragraphs could be converted into stanzas or something else. Again this all is depended on the poetry used to train the model.

Years ago I use the python NLP library to generate different poetic forms such as a sonnets or villanelles from a corpus like Shakespeare or the bible. I thought, what sort of interesting thing could I do with python NLP.   There are, for sure, a ton of boring, uninteresting and uninventive things I could do.

I went and did:

from nltk.book import *

And saw that Moby Dick was the first book included with nltk.  I read it a long time ago.  I also read Charles Olson’s,  Call Me Ishmael during my weekly sonograms while pregnant with my second child while stricken with a mild case of gestational diabetes.  This is a work of poetic literary criticism centered on Moby Dick, it is excellent and inspiring. Charles Olson is a character, a poet and teacher (perhaps one time president) at Black Mountain College , he wrote an poetic epic on Worcester MA that I own but have not finished.  A few years back, I  picked up an Olson bio from Canio’s in Sag Harbor. It was a great read. I love reading poet biographies!

This is a round about way of saying that  I want to use Moby Dick for my poetic experiments. In this current experiment I used all the texts included in the nltk. But maybe eventually I’ll move back and focus on Moby Dick.

The tools in chapter 1 of the online nltk book are: frequency, distribution, word length, colocation and bigrams (ie words that are often together). However  I have to do some weird stuff to get some of the functions to output to a list instead of stdout.

The first chapter also focuses on the issues with translation and ambiguity.    For example:

The people were found by the searchers vs people were found by the afternoon 

This represents difference senses of by. In Latin a temporal sense would use a different preposition and would probably be in the accusative I think, otherwise it would be in the ablative. But this aint Latin is it now!

My first poetic experiment creates a poem(s) from the included texts by alternating between high frequency short words and low frequency short words and low frequency long words. It is fun. This was one poem generated:

foul four woods
foul four woods
circumstances significance encountering, Nevertheless superstitious

four woods hanging
four advantage uncertain
accommodation circumstances respectable, inclination understanding

four second here
four second here
Philistines everlasting exceedingly, peradventure generations

four Until advantage
four advantage Western
contributed circumstances willingness, responsible remembering

woods second fingers
fingers here NOT
foul four Until
second here Three
four Until advantage
four advantage Western
contributed transactions Connecticut, complicated introduction

reliable music travel
music travel A
foul four marching
four yellow Does
considerable impatiently intellectual, extraordinary astonishment

A few things – I think this poem should just have four stanzas. I love the last line. I love the repetition of four. I added in the commas manually, but I think I have to consider punctuation in the generation of these poems. Also I am interested in how this creates new poetic forms. Instead of rhyme and meter, iambic pentameter and what not, we are thinking in terms of statistics. What is a poetic form based on statistics? This is sort of interesting.  Am I creating a poem, or a poetic form? Here is the first bit of code – I think even the source code is sort of poetic.

Finally, here is a NYTimes published this article about bell hooks and I highly recommend it.

NLP Python Day 2 – Catherine Bergvall

One of my favorite contemporary poems is Via by Catherine Bergvall. You can listen to it here, on ubuweb.

Via takes 48 translations of the first line of The Divine Comedy.  I find it breathtakingly beautiful, listing to and reading this poem is good for my soul.  I was thinking what would it be like to make 48 first line code re-interpretations of a text.  Here we are thinking about algorithmic remapping instead of creative translation. It is not what the poet is saying but the materiality of the poem.   What does this have to do with the poem  and what if we use this layer to create a new peom.  It is not a  translation but a remapping, a trans-mapping, or perhaps a transduction.

What if we looked at Moby Dick and generated 48 new lines? How would we do this? Would we generate the first line based in the whole corpus? This would not even look like a first line. Would we generate the first line based on the the first line? This could get old real quick.  Would we look at derivative works? Or could we look at pieces of criticism and use that? What about rearranging the first line – anagram style?

I eneded up doing something pretty bogus. Using the sent1 function that returns sentences and then iterating on the words to find similar words and similar contexts. This is the result.

have think say called in thought as will let tell me of that and saw
take see know to account
well all loomings me ll again i ye you him greenlanders the they an
significantly it the of and upon fishermen a mariners it should this
they my some him he me ashore a they them me the near all
him it them us you which all be queequeg ye that see say thee one this
her ahab and sea
upon at choked are in if to i with writing between and by hold
raises a it here for seek dam i startled look reads that pilot still
in an makes a tell such told a induced to turned to
what in it ahab did and hinted prophesy am bound see yet guess for had
except been to take goes
ha muttered i but on said man s yes the i was than can me some
thyself can now that go said now how i should here for been but
dear be hypo tell to are one bloody unlettered hope

It is not as poetic as Via. I think there is more I could do with this idea. I am going to something related to computational linguistics and/or statistics. Chapter 2 is all about importing other corpuses from Jane Austin to Reuters. One of the interesting things is time series data and how the usage of a word changes over time. It would be interesting to do a version of the first line of the Divine Comedy in this way.  A sort of evolution through time, rather than through translations (although this too is through time).

There is also a review of related tools, lexical relationships foreign languages, stop words. lemmas (synonyms), the word net hierarchy, which is a tree of words and their relation.  For this poem above. I could probably do more with lexical relationships but I also want to experiment more with the structure and like I said before – stats!

Ooops – bug in day one of NLP Python

There was a bug in my code for first day NLP python, which I realized when I started doing my next experiment.

The text that was used was only Moby Dick, but the frequency distributions came from the individual texts. So if you look at the poem you will see that the reason why the stanzas are different is that the words selected by frequency distributions are from different corpuses BUT the words themselves are all from Moby Dick.  This ended up working.

This new poem pulls both the new distributions and new texts from the different texts:

text1: Moby Dick by Herman Melville 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908

It is not so different – maybe there is still a bug. It gets a bit weird in the middle which I like. I am not so sure the last poem was a mistake but a happy accident. It is an interesting idea to explore- taking frequency distributions from  other corpuses and using them on different lexicons. Maybe I will explore that more.

foul four woods
foul four woods
circumstances significance encountering Nevertheless superstitious

four woods hanging
four looking eligible
endeavoured recommended Somersetshire respectable acquaintance

Leah four hath
Leah four hath
Peradventure circumcised peradventure buryingplace everlasting

four aegis Until
four looking Western
contributed circumstances accomplishment acquisition willingness

woods lord Elev
fingers here kids
#14-19teens ))))))))))))) #talkcity_adults Compliments )))))))))))

four Until Winter
PERSON Go MAN
BRIDGEKEEPER

Heights four railing
four Western regional
contributed standardized headquarters substantially acquisition

children outings Non
children Seeking to
relationship

yellow four Does
yellow four Does
information considerable impatiently intellectual astonishment

Cryptocurrency Trading Bot

A bazillion years ago, when I worked at Morgan Stanley. There was a big sign above the entrance and exit to the trading floor:  “DID YOU MAKE MONEY TODAY?”  If you are in finance that is the only metric that matters.

This summer I wrote a paper trading bot to arbitrage bid/ask spreads across different crypto exchanges. Here is the git repo – it is in progress and there is one glaring bug so if you run it you will lose $$.

So lets talk a little bit about what is going on here and how these exchanges work.

Well someone offers to buy 3 bitcoin for $100, someone else offers to buy .5 bitcoin for $105 and so forth. This builds one side of the order book, and the different quantities and prices represent the depth of the order book.

Then there is the other side of the order book, someone offers to sell 2 bitcoin for $200, another person offers to sell .5 bitcoin for $195.

The trade is made when someone else offers to take the other side. If I come in and say yes! I will buy 2 bitcoin from you for $200. Bam the trade is made and that is removed from the order book.

Here is an example json of the order book from hitbtc, an exchange where you can trade crypto.

Now the bid/ask arbitrage is when you can lock in a profit. When on one exchange you can buy 2 bitcoin for $200 and immediately sell it for $205. This will not happen on the same exchange but it can happen on different exchanges, so if you have a script that is constantly monitoring a bunch of currencies and a bunch of exchanges.

This does not happen in regular exchanges for normal people like me, and probably you if you are reading this blog instead of something on your bloomberg terminal. These types of trades happen very quickly. For traditional exchanges, you want to be as close as physically possible to the exchange, and so forth.

For the crypto exchanges, there is no physical exchange, except perhaps for Gemini, which I think has a physical exchange. So there is no direct pipe to the exchange. The reason why you can arb in this way is because of the Risk.

So this is the other big idea in trading RISK! What kind of risk are you taking, and what is your risk versus your reward. Theoretically if you are executing the trade instantaneously then what is the risk?

As Milton Friedman said – There aint no such thing as a free lunch.

Aint that the truth! When you are making all this money on this supposedly risk free arbitrage you are still taking on risk but what is that risk?

Well in the case of these crypto exchanges you have exchange risk. The exchange may shut down, may be hacked, you can lose all your money. That is the big risk and you are being compensated by it with the arbitrage trade.

There is also currency risk. If you trade your crypto into some sovereign currency (USD, EUR, etc), there is a high transaction cost and you will lose money. In order to make money you have to keep your money in crypto and maybe exchange into USD once a day. This means you have exposure to the directionality of the crypto. You may make money on each trade, but if crypto goes down one day it does not matter because you will have lost money relative to the dollar.  You can probably mitigate this with options, but it is something to consider.

Other things to consider are: the percentage the exchange takes on each trade, slippage, that is if you are unable to execute at the price you want, if the market moves and you can only execute one side of the trade and so you have to unwind the trade of keep the position on. But these are run of the mill trading issues. There is nothing special about it being in crypto.

This is interesting. I have been looking into deep learning algorithms for predicting crypto prices but it is difficult because time series data is autocorrelated. Each tick or trade is based on the previous trade. I want to look into this aspect in greater detail. I am not interested in numerical methods.

Decentralized Protocols

This week I worked on some scuttlebutt with Tenor and Ben. Although I building the prayer project on Ethereum, itself a decentralized protocol, I never looked too much at the general landscape of decentralized protocols… until now.  To get myself up to speed I listen to the DAT podcast, a third web podcast on scuttlebutt. This second on was an interesting podcast where the founder of scuttlebutt (sea slang for rumor), distinguishes between cyberspace (the space of communication) and cypherspace (the space of algorithms).

I downloaded Patchwork (a social network built on scuttlebutt) then I ran a patchwork  server. I thought maybe I should build a prayer sb tool, and so I looked at this chat code.  After listening to  Francis‘ talk at Recurse and read a bunch of papers and visited sites. I started going through the proto school exercises which I did not know were made by protocol labs that includes things like ipfs, Nomad and raft . I downloaded beaker, which uses dat, not scuttlebutt. AND I saw great fantastic animation of how raft works – I suggest you watch it if you are interested.

The decentralized tech I was familiar with before was blockstack. They have a great git repo, but looking at all these other initiatives has really opened by eyes.

A gossip protocol is a way to describe how peer to peer nodes share information.  Nodes have a rule  that determines what messages the believe. This rule is the consensus algorithm (I think). There is also no guarantee that every node will receive every message.

So the question is, do we think that bitcoin and ethereum are gossip protocols???  According to the internet it is unclear. Gavin refers bitcoin as using a gossip protocol. Other people do not. The bitcoin and ethereum protocols are heavier than just a gossip protocol – they consensus built in. It seems that gossip is not about consensus.  Scuttlebutt is not looking for universal consensus. Each node controls what truth is. But that consensus is layered on top of a gossip protocol. Gossip is not interested in agreeing with other nodes, but it making a decision for itself.  But I am new to this so I may be off base.  The difference appears to be with the topology of the network. Everyeone

There is an idea of a blockchain like system. Scuttlebutt uses Kappa Architecture, where all activity is appended immutably to a log file. However it is not the log file that all nodes must agree on but the message … I think…. I could prob

After going through scuttlebutt, it seems like a useful architecture for lightweight decentralized communication and decision making but not right for the prayer project. One of the things I am interested in with the prayer project is the notion of action or execution of a prayer. Scuttlebutt is about communication and sure we can build something on top to execute, but I like coopting the logic of exchange built into currency for another purpose.