A Naive Experiment in Programmatically Optimizing DraftKings NFL Lineups: Part 1

DraftKings is a daily fantasy sports (DFS) sports site that has awarded millions of dollars of payouts every season and enticed millions of players to join and deposit, all under of the premise of believing they are “smarter than the average fan.”  I set out to test how good players are at maximizing player value within the salary cap, compared to an algorithm.  The test had many shortcomings, but in 10 free Week 1 contests, the algorithm’s average finish was the 40th percentile which is promising for winning 50/50 and H2H contests.

Continue reading A Naive Experiment in Programmatically Optimizing DraftKings NFL Lineups: Part 1

First Ever Interactive State of the Union

Live stream content concurrently with sidebar content.
Live stream content concurrently with sidebar content.

I really think the White House accomplished something today by live casting the State of the Union address on YouTube with sidebar content and simulataneously posting graphs, quotes and pictures on www.whitehouse.gov

Not only do I find this just generally effective, but I was interested to see one of the interactive poll results.  Over 1/3 of respondents were suburban, which indicates to me that people engaged by this new interactive media is broader the just high-tech city slickers.

And since I can’t forget that I’m a liberal, don’t miss my video of John Boehner’s mysterious lips.

Poll results
Poll results
Simultaneously watching the SOTU on Youtube with sidebar content and companion tablet content.
Simultaneously watching the SOTU on Youtube with sidebar content and companion tablet content.



Making PDF Newsletters Accessible

As the Rhodes 19 Fleet 5 webmaster, I was faced with doing something with over a decade of PDF newsletters. I decided the best thing to do was to extract and structure the content, then put it into WordPress so that I was easy to find, access, archive and index.

The newsletters were produced in Word, then saved as PDF and used multiple columns, making it basically impossible to do this in a completely automated way. But, there had to be something than adding the content post-by-post into WordPress.

Complete Automation? Nope.

All of the content was stored in the PDFs as text, so I figured it must be possible to extract it. So, I used PDFSam to merge them into a mega-PDF, then trying producing HTML with c:\bin\pdftohtml.exe -i -noframes -nomerge all.pdf and voila I had a giant HTML file. Unfortunately, this is where I abandoned this technique. It didn’t pick up headers and instead just made them <b>Heading</b><br/>, which wouldn’t do. Not only that, but it left everything riddled with <br/> everywhere.

Semi-automation? Yep.

I wondered “If I structure the content, would it be easy to get into WordPress?” Some quick Google-ing turned up WP All Import which could take XML and create posts. Sweet.

So, if I could get the content into XML, then it would be easy to that into WordPress. And that is just what I did, using the following format.

<?xml version="1.0" encoding="iso-8859-1"?>


Taking print formatted content (e.g. with columns) and turning it into web formatted content is about as hard as it sounds. Relying on software to extract the content with human intervention is unreliable at best. Even if there was software that could use a user-defined template to know where to look for stuff, then I’d be at the mercy of 10 years of class presidents using slightly different formats. So, my final solution seems like a reasonable balance of simplicity and automation.

  1. Install the plugin WP All Import
  2. Structure the PDF content as XML
  3. Import the XML with the plugin.

Note: At first I wrestled with UTF-8 for a bit before landing on iso-8859-1.

Open Ocean, a Reef and the VO65 That Hit It

Tracks of the other competitors showing just how close other boats came.
Tracks of the other competitors showing just how close other boats came.

Team Vestas crashed into the Cargados Carajos shoal in the middle of the Indian Ocean while going over 19 knots in the middle of the night (1510 UTC) on Saturday, November 29, 2014.  Everyone is safe, so now we can begin to comprehend what happened.

For my non-sailing friends: hitting something at night is the ocean racer’s worst nightmare; it is the equivalent of hitting a jersey barrier going 25MPH in your car without your headlights on.  The footage, photos and coverage are legendary and unprecedented.  Everyone I know and myself included is in disbelief.

I will add a link to expert commentary about the navigation error once its available.  Read this post by scrolling down to the pictures and video below, then if you want to learn more, try some of the links.

Update at 9:19AM EST 12/7/2014: More and more responses are being written so I’ve updated this post a little bit to reflect that.

Assorted links:

Video of the Crash

See similar footage on YouTube or read more at Sailing World.

Map of Cargados Carajos Shoal

Follow Up Footage

More Photos

Vestas on the Reef
Vestas damage and scale.
Vestas on the Reef
Vestas on the Reef
Aerial view of shipwrecked Vestas
Aerial view of shipwrecked Vestas

Using Google Contacts on Android

my contactsI finally have my Android contacts synced in a way I like with my Google Contacts and it only took 2 years.  I wanted a single, unified contacts tool with photos.  I didn’t want to have to use a 3rd party app to “automagically” manage my contacts.  I didn’t want to permanently or irreversibly link my various contacts stores and risk polluting the data.

All My Contacts in One Place: Google Contacts is the only tool I use for storing contacts on my Android phone and I’ve taken care to merge duplicates, reduce noise and add photos.  Here are so tips:

  1. Accounts Do not let your phone store contacts on the device; force it to use your Google account.
  2. Duplicates Do not use “join” from your phone.  Use “Merge” from the web interface.  The prior (join) only affects the contacts on your device; the latter (merge) affects your whole account.
  3. Pictures I loaded my Facebook friends pictures into my Google Contacts in a controlled manner using “Contact Pictures Sync
  4. Noise I reduced the number of “contacts” I had from 1200+ to ~400 by migrating contacts out of “My Contacts” into “Other Contacts” from the web interface.  “Other Contacts” are kept around for autocomplete on email (at least from the desktop interface) but don’t clutter up the “Contacts” that show up on my phone.

I couldn’t find a guide to walk me through this, so I figured it was worth documenting.

I Fought Comcast and Almost Won

Comcast has a bug involving their online customer portal and financial back-office that results in customers who enroll in Ecobill receiving no bill whatsoever.  It’s documented all over their forums 1 2 3 4 5 6 7 8 9.  I experienced this bug, and once I discovered there wasn’t a quick fix, I thought I’d try to “help” and report the bug.  At this point I’ve actually managed to talk to a non-customer service human at the Comcast Northeast corporate office (I am so proud!), but still haven’t been able to successfully report the bug.  I’m sort of proud of this, since everyone hates Comcast and wants to give them a piece of their mind, so below I’ll explain the bug, and the process.

EDIT: There is some discussion here on HackerNews, which has brought in around 5,000 visits to this post.

What Causes the Bug?

While only a Comcast software engineer can truly determine this, I have some partially validated ideas.  It’s quite likely that Comcast has at least two databases: the one that the online customer portal (OCP-DB) is hooked up to and the one the financial back-office (FBO-DB) is hooked up to.  We’ll assume this is true for now.  The OCP-DB is the one customers can touch, and contains mostly things like preferences (think “Forward my email to my gmail” etc.)  The FBO-DB is the one that only the finance department can touch, and also the one that some electronic bill printing tool polls to determine who to print a bill for.

So now you’ve got customer data stored in two places, and you need to sync them.  Syncing can actually be a challenge: Which database is the “master”?  How often should they be synced?  Sync every record or just altered records?  What to do in the case of a sync failure (power outage/connectivity issue/etc)?  Whatever the Comcast software engineers chose, appears to have a bug.

Users report that they that:

  1. Enroll in Ecobill
  2. See in the OCP that their “Bill delivery preference” = Ecobill
  3. Do not receive a paperbill or an Ecobill

So it appears that the OCP-DB says “status = ecobill” and the FBO-DB got synced to set “deliverPaperBill = false.”  However it seems that it is missing some flag that says “sendEcobillNotificaiton = true,” so the customer gets no bill at all!

How Did I Actually Get to Speak to a Human?

So this started in July, but the first couple of times I just said “Eh, must just be a temporary flaw” or “Maybe I missed the email.”  It wasn’t until the New Year that I got fed up and said “Let’s deal with this.”

  1. Chat with Customer Service
    I started with an innocuous chat with customer service.  I had a total of 154 back-and-forths with 3 different reps before I threw up my hands in disgust.  Frustrated as I was, this is where I discovered that their must be two databases because two different reps said “let me check the system – you are not enrolled in Ecobill” despite the fact I was looking at the “Bill Delivery Prefernce” line that said “Ecobill.”  They didn’t solve my problem, but they did make me even more angry at Comcast.  One of the reps asked me for my account credentials in plaintext.  No customer service rep should EVER ask for that; with them they can modify your account as they please (i.e. add additional services, etc) and since they’re acting as you there’s no approval required.  I infer that he was thinking “This guy is an idiot.  I will just log into his account and fix it for him.”  Who knows, but fuck that.
  2. Start Hunting the Forums
    “Comcast customers have viewed over 500,096,961  pages in the Help forums,” so it seemed worth a try.  What I found is a forum, moderated mostly by 2 actual Comcast employees.  That’s right, it appears that for the whole nation, Comcast has only 2 main forum moderators.  Luckily there are lots of non-Comcast semi-moderators who help people.  Anyway, after some searching, I discovered that: Yup, other people have this problem.  Nope, there is no fix.
  3. Track down some Email Addresses
    Realizing that these 2 poor forum guys had larger issues, I started hunting for email addresses, and found one for some Comcast executive boasting to have “23 years experience with Comcast products.”  I emailed him, and lo-and-behold, two days later he actually emailed me back and said he’d escalated appropriately!  OMG!
  4. Talk to a Human
    Yesterday I got a voicemail from an unknown number.  It was someone from “Comcast Northeast Corporate Office” calling me about my billing issue.  Wow!  We finally spoke today.  We covered several things:

    1. Sorry They were sorry I was having this problem
    2. Explanation There are two systems at Comcast, and sometimes they get out of sync.  Huzzah!  Confirmation of my hypothesis!
    3. Acknowledgement They are aware customers aren’t getting bills
    4. Indifference They’re not going to fix it

So, What’s the Resolution?

Well, basically it sounds like Comcast knows there’s a bug, but they’re not going to fix it.  In some strange irony though, I think I solved the problem myself.  At some point in all this, I re-enrolled in paper delivery, then re-enrolled in Ecobill.  This wasn’t trivial.  I didn’t want a paperbill; I wanted an Ecobill, but on the preferences page it said “Delivery Preference” = Ecobill, and the save button was disabled, unless I instead chose a paper bill, then I could save my preferences.

Today I got my first Ecobill.

It was for the wrong amount.

Fuck my life.

In Boston Mayoral Race, Voter Turnout by Ward Different from Preliminary Election

Today, President of MassINC Polling Group Steve Koczela posted a graphic of voter turnout by Ward for the Boston Mayoral election.  The tweet is copied below.  Note that the turnout is up significantly in many wards, compared to turnout in the preliminary elections.  There are about 372,000 registered voters in Boston.  The wards are laid out as designated in this PDF map of wards and precincts (cityofboston.com/maps)  You can information about the election at WBUR’s Election Day in Boston page.

Percent of Electorate from Each Ward

turnout by ward

Further Reading

Commonwealth Magazine’s “Where the voters are in Boston” has more coverage of voting in Boston.  In the last 30 years, voter representation in some wards has more than double; in other wards it’s down as much as 40%.  They write:

In 1970, almost 38 percent of Boston’s residents were under the age of 18. Today, children under 18 account for just 17 percent of our city’s population. Furthermore, the average household size has decreased by almost 25 percent in the last 50 years.


On the Future of Team Racing

Yesterday Stanford University assistant sailing coach Clinton Hayes claimed that American team racing needs a new dinghy, more spectator participation and innovation in his piece The Future of Team Racing.  He writes:

“One reason why people love team racing is because you don’t have to own a boat. They have the time of their lives racing in college fleets and want to do the same post college. Combine this with a tough job market and little free time and its no wonder why no one wants to own a boat. Traditionally, the Vanguard 15 was the boat to get after college. V15 team races often drew 20+ teams and fleet racing nationals over 100 boats. There is still solid participation but less than half of what it once was. For team racing to grow it needs to give the people what they want!”

I couldn’t agree more and I think the most critical piece is the social aspect.  In other post-college sports, when jobs and new realities get in the way of maintaining peak physical condition and practice time, the most important element becomes camaraderie.  Players show up for games and compete against other teams, but a big part of the fun is shooting the sh*t on the bench, grabbing a beer/bite afterwards and traveling to and from the venue together.

Any changes to the racing format should keep this in mind.  In my mind this means the following, many of which Clinton covered:

  • The boat needs to be durable, so that team racers don’t have to spend much time off the water making repairs and so that the boats remain cheap (either for private or club ownership.)
  • Knockouts are great, even though it means the bottom tier teams aren’t competitng at the end, since they are sticking around and cheering on the top teams.
  • Making the race course accessible for spectators is critical.
  • Maintaining “pick up” events like the various random pairs team events will become increasingly important as teams get better and better.

Big Boi at Paradise Lounge

bigboi3Last night I witnessed an amazing show by Big Boi with performances by Killer Mike, Sleepy Brown, Real T@lk and Renegade El Ray.  My ears are still ringing.  The set list was sick.  The energy, particularly from Killer Mike, was unreal.

If you’re still saying “um, what?” you need some Hip-hop edumacation and I advise you to go back to your childhood and remember some big Outkast songs like “Bombs over Bagdag” or “Ms. Jackson,” and checkout out “Sir Lucious Left Foot: The Son of Chico Dust.”


  • Killer Mike

    Killer Mike’s set was incredible.  With the help of his booming voice and quality lyrics, I really felt the passion with which he loves playing music.  He brought himself to tears playing “F*ck Ronald Reagan,” a song about the way he thinks American politics shaped the country as he grew up in the South in the 1980s.  Then to end his set, he got into the middle of the crowd and looked us all in the eyes as he sang about the incredible joy brings to his life.  Finally, he made everyone laugh as he walked offstage and said “Thank you Boston, Ima go smoke a blunt and watch Breaking Bad.”

  • Inches from the Stage

    We lucked out and got to stand first row, and I often forget how incredible being close to a performer is: it’s insane.  We could have touched Big Boi, we did touch Sleepy Brown and twice Killer Mike got down low and rapped right in our faces.  Unreal.

  • Big Boi’s Bum Knee

    Big Boi was playing with a bum knee, which he injured in May and which caused him to postpone his June show.  To compensate, he had an ornate, larger than life, “King’s Chair” where had to frequently sit down and adjust his knee brace.  It gave the show a raw, awesome feel.


Set list, photos and some video below.