#OpenData just got a teeny bit more open

A map showing real-estate development projects as of the end of 2012. Data visualization from the city's Open Data Portal.

We were disheartened when, after submitting some fairly innocuous questions to the Mayor’s Chief Innovation Officer, Jay Nath, we received zero answers. By the time the Guardian’s annual Freedom of Information issue hit stands yesterday, we were still out in the cold. (Shameless plug: Pick up a print edition of this week’s paper for our flow chart on how to file Sunshine requests, designed by our illustrious Art Director Brooke Robertson.)

Nath, who helped start the city's Open Data program, responded to our emails and tweets (apologetically) by saying he was awaiting the green light from the Mayor’s Office of Communications. Which begs the question: In a city so outwardly committed to transparency, why can’t the Mayor’s Office of Communications entrust a program expert to share information about information-sharing software?

Anyway, the day after we ran our story, Nath did respond in an email. The first objective of Open Data is to “increase transparency,” he told us.

Other goals are to “drive economic development” and “foster the creation of new services and analysis by our community.” The inspiration behind it came from President Barack Obama, who on his first day in office “issued a memo on open government that heralded their open data program Data.gov," Nath explained. "With this precedent, the city recognized an opportunity to share local data with the public.” 

Head over to the city's Open Data Portal and you can poke around for info on everything from real-estate development, to restaurant health inspection scores, to city salary ranges by job classification.

As Nath pointed out, there are also over 30 datasets around campaign finance. That’s a good thing – but there’s still room for improvement. Last year, after attending a city hackathon where transparency advocates hoped to spur creation of an app to track lobbying, campaign contributions and real-estate development, Adriel Hampton of the San Francisco Technology Democrats noted that this was impossible due to a lack of information. “Despite millions in spending on … online transparency measures, access to data in these areas is woefully lacking,” Hampton wrote.

Nath said the annual cost is $40,000 per year for software. He also shared his vision for future expansion. “In terms of new services, I see applications that mash up data from multiple public and private sources to create a seamless experience,” he said. “For example, imagine a tourism app that helps you navigate the city via public transit, taxis, car / bike sharing, biking, walking, etc.”

So how does Open Data affect public records requests under the San Francisco Sunshine Ordinance? “Government can use open data to reduce costs by pro-actively providing information that is often requested through FOIA,” Nath told us, referring to the Freedom of Information Act. “For example, by releasing real-time transit data, transit riders have dozens of ways to know when their next bus is coming. This new and immediate access to information has resulted in 21.7% fewer SF 311 calls – and at $2 per call – that yielded a savings of over $1 million a year.”

An interesting thing about data is that it can be totally neutral until it’s harnessed for a particular purpose, with clever visualization and presentation. Just ask the producer of this video on wealth distribution, which has been making the rounds.


significant margin in the last election, can you really blame his staff for being less than totally co-operative with you? You certainly no longer wonder why so many of your phone calls go unreturned.

Perhaps if you took a more balanced and nuanced view of local politics, doors would suddenly, magically open? Ya think?

Posted by Guest on Mar. 07, 2013 @ 12:56 pm

That SF only stands to benefit from a diversity of perspective and healthy democratic debate.

Posted by rebecca on Mar. 07, 2013 @ 1:47 pm

leads to allegations of "trolling" from those less enlightened than your good self.

Posted by Guest on Mar. 07, 2013 @ 3:06 pm

When the "criticism" consists of unsubstantiated claims, ad hominems, personal attacks, assertions and inflammatory statements that only wish to bait --with Lucretia and several "guests" admitting they are just trying to provoke people here--I do not think the charges of trolling are off the mark.

Posted by Guest on Mar. 07, 2013 @ 8:08 pm

The hypocrisy of your comment is that you are a daily resident of the "house view." (You and the other right-wing trolls). You live on this site every day of the week just so you can disagree with its views on every topic and express your hate as a right-wing nut. What mature, well-adjusted person would do that or has the time to? But that's the reason you're called a troll because of the hateful and smug content/style of your posts.

Posted by Guest on Mar. 07, 2013 @ 8:12 pm

commenter for his or her succinct analysis of the downfall of these comment pages as a forum for discussion among thoughtful readers.

Down with stupidity!!!

Power to the thoughtful!!!

Posted by San Francisco Anti-Stupidity Campaign on Mar. 07, 2013 @ 8:39 pm

Your own post refutes that view.

Posted by Guest on Mar. 08, 2013 @ 10:55 am

"Given how hostile SFBG is to our Mayor, even though he won by a significant margin in the last election..."

Your savior won by 9%, pleb.

Why are you STILL campaigning for savior, pleb? The election is over. He got 30% of the vote in the first round. It took him 12 (twelve) rounds to get to 59%. We went over this once before, pleb. Are you that dense? No attention span at all?

59% is hardly a "significant margin" that you like to gush over. You're coming off as a loon amateur troll, as usual. If your savior Lee were the choice of the majority of those who voted in that election---which he clearly was not---he would have received 59% in the very FIRST round, pleb, regardless of the number of candidates. So you can shut off your gushing for your savior because you're not fooling anyone. I didn't realize his campaign office was still open. And I didn't vote for Ávalos so you don't need to drag out the Ávalos Card.

I never did get an answer to this and you would know: In the Church of Lee, is it true that Rose Brown and Willie Pak are acolytes and do you use incense in your liturgies? Please get back to me before visiting hours are over there in your ward.

You also dumped this excrement:

"Perhaps if you took a more balanced and nuanced view of local politics, doors would suddenly, magically open?"

Translation: Cheerlead for my savior at every opportunity and for conservative politicians posing as so-called "moderates."

International Troll Society Member #12360969212

Posted by International Troll Society Member #12360969212 on Mar. 07, 2013 @ 7:40 pm


Your savior won by 8%, pleb (since 51% of the vote is required for "victory" to be declared).

8% is a "significant margin" to you, pleb? What podunk school did you go to?

International Troll Society Member #12360969212

Posted by International Troll Society Member #12360969212 on Mar. 07, 2013 @ 7:56 pm

59% is nearly 50% more than 41%.

Call it three to two, virtually a landslide.

Posted by Guest on Mar. 08, 2013 @ 10:57 am

You still do a print edition?

Posted by Chromefields on Mar. 07, 2013 @ 1:09 pm

Without all those hooker ad's, it's just a whole lot of nothing.

Posted by Guest on Mar. 07, 2013 @ 1:44 pm

We should be able to browse all public records online, properly coded for legal exclusions, without having to bother staff to service sunshine requests.

Posted by marcos on Mar. 07, 2013 @ 1:10 pm

(and I find it inconceivable that you have not) then you will know that the first three rules of discovery are obfuscate, obfuscate, obfuscate.

Send the other side's lawyers 150 packing boxes full of paperwork containing everything EXCEPT what would really hurt you, which of course gets "disappeared".

You run up the costs of your enemies, while giving away nothing, and usually a settlement results.

You have so much to learn.

Posted by Guest on Mar. 07, 2013 @ 1:23 pm

Are you suggesting that the government should intentionally adopt an adversarial, hostile posture towards citizens and voters?

Posted by marcos on Mar. 07, 2013 @ 1:38 pm

being a pesky troublemaker who likes to cause trouble but adds little value, then it should not comes as a huge shock to you if and when people become a tad less co-operative.

Try diplomacy rather than a blunt instrument.

Posted by Guest on Mar. 07, 2013 @ 1:46 pm

Departments operating in secret is aggression in the first instance.

Posted by marcos on Mar. 07, 2013 @ 2:19 pm

I feel sure that the city is willing to open their doors and drawers.

But if, hypothetically speaking, you have historically been a right royal PITA, then it would shock nobnody if you fail to garner the co-operation that you might otherwise obtain.

Thing being - your attitude matters. Is yours kosher and honest? Or skewed and self-serving?

Posted by Guest on Mar. 07, 2013 @ 3:12 pm

unless they have some sort of plan to follow.

This one is a riff off of Big Troll Lie No. 17: "only whiners show up at public meetings"

Posted by lillipublicans on Mar. 07, 2013 @ 3:39 pm

I simply noted the inevitable human implications that ensue when someone proves themselves to be troublesome and then they exclaim that somehow, mysteriously, magically, they are not getting the co-operation that ideally they might wish for.

Attitude matters.

Posted by Guest on Mar. 07, 2013 @ 4:04 pm

The law is the law, and there is no accounting for the purpose of the request or the requester in the law.

Posted by marcos on Mar. 07, 2013 @ 7:30 pm

Anyone who has worked in any kind of job knows that attitude matters and affects the outcome.

Posted by Guest on Mar. 08, 2013 @ 10:56 am

up your boss' ass that you can't see anything but butt cheek, and that that is how you have distinguished yourself in the workaday world.

Posted by lillipubicans on Mar. 08, 2013 @ 11:12 am

cop an attitude when dealing with other people upon whose co-operation they rely, do then risk that the outcomes will not be favorable to them.

I always get the information I want from the city. If you do not, that probably tells us more about you than it says about the city.

Posted by Guest on Mar. 08, 2013 @ 11:17 am
Posted by lillipubicans on Mar. 08, 2013 @ 11:30 am
Posted by Guest on Mar. 08, 2013 @ 11:41 am

behavior would also be running up the costs for his/her client. You must have had some really bad lawyers, clown.

Posted by Guest on Mar. 07, 2013 @ 8:01 pm

distracts them from the real dirt.


Posted by Guest on Mar. 08, 2013 @ 10:58 am

and costs that can be billed to a client?

Posted by Guest on Mar. 09, 2013 @ 12:11 pm

The City lost the last sunshine case brought against it and had to pay $25,000 in attorneys' costs.

Posted by marcos on Mar. 09, 2013 @ 12:24 pm

taxpayer, you have to pay the costs of defending that.

Posted by anon on Mar. 09, 2013 @ 12:48 pm

The easy answer is for the bureaucracy to follow the fucking law and save the taxpayers money.

Posted by marcos on Mar. 09, 2013 @ 1:11 pm

Now you don't?

Which is it?

Posted by anon on Mar. 09, 2013 @ 1:47 pm

Even if you a plaintiff with a contingency lawyer, you will still have to pay out-of-pocket costs such as filing fees, depo's, and other discovery costs.

So even if you win, you still have had to pay those. If you lose, of course, you get nothing and have still paid those.

But, as noted, the real purpose of over delivering is to snow the other party's lawyers, obfuscate and disguise any real evidence there (assuming that it wasn't conveniently destroyed anyway) and generally place an intolerable burden on the other party. Works particularly well against small firms.

Sure, you can quote me CCP rules that, of course, theoretically discourage such tactics. But you're still in law school if you think it doesn't happen in the real world. It's routine.

Posted by anon on Mar. 09, 2013 @ 12:30 pm

by anon. You slipping up there, anon? Or is it Enrique Pearce?

Posted by Guest on Mar. 10, 2013 @ 6:27 pm

I should know. I do the billing for these shysters.

Posted by Guest on Mar. 09, 2013 @ 12:46 pm

Did they find that in some corporate marketing manual? Here's an innovation for you -let's fire all the patronage parasites and use the money for something useful.

Posted by Greg on Mar. 07, 2013 @ 9:21 pm

using other peoples names.

Everyone knows that the real Greg loves all of these useless government employees and commissions.

Posted by matlock on Mar. 09, 2013 @ 5:56 pm

Not to mention Asians and cops.

Oops, I just did.

Posted by Guest on Mar. 09, 2013 @ 6:08 pm

"The inspiration behind it came from President Barack Obama, who on his first day in office “issued a memo on open government ..."

I guess that memo stated that he (Obama) wasn't going to have one (an open government). Because Obama has been more secretive than Bush. This article (link below) is from 2010 and it's only gotten worse since then:

Another broken promise: Obama more secretive than Bush

Posted by Guest on Mar. 07, 2013 @ 9:40 pm

Submitted into evidence for Obama's upcoming war crimes trial:

Protect Whistleblowers: Often the best source of information about waste, fraud, and abuse in government is an existing government employee committed to public integrity and willing to speak out. Such acts of courage and patriotism, which can sometimes save lives and often save taxpayer dollars, should be encouraged rather than stifled. We need to empower federal employees as watchdogs of wrongdoing and partners in performance. Barack Obama will strengthen whistleblower laws to protect federal workers who expose waste, fraud, and abuse of authority in government. Obama will ensure that federal agencies expedite the process for reviewing whistleblower claims and whistleblowers have full access to courts and due process.


Posted by marcos on Mar. 08, 2013 @ 7:27 am

If the government can ignore open data laws, then what other laws is it allowed to ignore?

Posted by marcos on Mar. 08, 2013 @ 10:39 am

but there is no law that says the data has to be organized or easy to analyze.

Posted by Guest on Mar. 08, 2013 @ 10:59 am

Public records have to be provided in their original format.

Posted by marcos on Mar. 08, 2013 @ 6:53 pm

Obviously, the city is not complying with the law, just attempting to make it look like it's being open and transparent. Somebody should call Nath's bluff.

Posted by Guest on Mar. 09, 2013 @ 12:52 pm

all of SF who give a flying crap about this.

Voters do not like to see their tax monies being wasted in providing fodder for troublemakers.

Posted by anon on Mar. 09, 2013 @ 1:14 pm

Voters like to see "troublemakers" make trouble for corruption.

Posted by marcos on Mar. 09, 2013 @ 1:30 pm

people being rewarded for doing anything other than vote. I vote to elect the kind of city leaders who will not listen to self-serving activists, but rather listen to the silent majority who do not endlessly whine like you do.

Posted by anon on Mar. 09, 2013 @ 1:49 pm

Democracy is a spectator sport. You get to vote every four years, and the rest of the time you should just shut up and go to work so that corporations can make some money. Leave the decisions in the hands of responsible authorities, unless of course those authorities happen to be progressive.

Posted by Guest on Mar. 09, 2013 @ 7:11 pm

And yes, I trust those elected officials like Lee who win with a clear voter mandate to do the right thing.

Posted by anon on Mar. 09, 2013 @ 7:43 pm