Update on Pulling Data from Discogs

Donnerstag 01 Dezember 2016

I initially ran into problems pulling the data from discogs because I was using a package that provided a Laravel implementation of cURL called Laracurl, and it didn't provide a header that discogs needed. So I made a change to the package and got it working. After someone advised me that Guzzle was a better package than cURL I switched to that, and in the process rewrote my matching code. After running the new, improved, streamlined code I have now matched all but 250 of my records to the discogs data, and 50 of those unmatched records are white labels which may not be matchable. So I am pretty happy with that ratio and will be moving on to my next project now.

Almost all of my records now have a link to discogs and a thumbnail pulled from discogs in the record detail page. If anyone wants to see additional info on the records such as tracklisting, year of release, or anything else, that info will be available on discogs. I didn't see any need to duplicate that data locally.

The code I wrote to automatically match my records to discogs did not match everything 100% accurately, and I tried to review the matches to make sure they were accurate, but I may have missed a couple here and there. 

Thanks to Discogs.com for making a great API. 

Etiketten: coding, music
Keine Kommentare

Discogs.com is one of my favorite web sites. I discovered it 15 years ago, and have used it since to research records and music and such. I just recently discovered that they have an API, so I used it to search for each record in my collection, and if it found results, I imported the link to the discogs page as well as a link to the thumbnail images to my database.

The process was rather convoluted. To start I just did a search for the data as it was in my database - artist, title, label and catalog number. But discogs often has multiple entries for each record - maybe it was released in different countries, or re-released, or has different entries for promos, test pressings and white labels. So my starting algorithm was as follows:

  • Match the full data in my database to a search request to Discogs API.
  • If one and only one result is returned, take that one and link it.
  • If more than one result is returned, filter out the ones for promos, mispresses, white labels, etc.
  • If there is still just one result, use that one. Otherwise mark the number of results returned in the database and move onto the next record.

This matched a couple hundred out of the couple thousand records in my database. Most of my records got 0 matches to Discogs, some still had multiple matches - anywhere from 2 to 35. So I started reviewing the ones with multiples by hand, and I realized that for most of the records with under 5 matches the matches were pretty much equivalent. So for those I just took the first match and assigned it. This matched another couple hundred records.

Now I put aside the few remaining records with from 5 to 35 potential matches and focused on the thousand or so that had no matches. Reviewing some of them manually, I found that many of them were due to typos in my database. So my next step was to omit the artist field and just check the title, label and catalog number. I got another couple hundred matches using this method. Then I went on and just searched using the catalog number. This method matched about half of the remaining unmatched records - but I had to manually verify each match because some catalog numbers are not unique. 

Unfortunately I do not have a catalog number for every record in my collection, and as of now about 1/3 of the records in my database are still unmatched. For those that are matched, on the record information page you will now see a link to the discogs.com page for that record, as well as a thumbnail pulled from discogs.com if available. For anyone interested in collecting records I highly recommend discogs.com as it is by far the most comprehensive database of music releases I know of. 

Etiketten: personal, music
Keine Kommentare

Democracy for Realists

Mittwoch 02 November 2016

This election in the US has got me thinking a lot about democracy and how it works, or in this case, doesn't seem to work too well. I get the impression that people don't choose their candidates based on the candidate's policy positions matching their own, but the opposite - they choose their policies based on which candidate or political party they support. Well I just read this book, Democracy for Realists, by C. Achen and L. Bartels, which confirms my fears and goes far beyond that to totally demolish what they call the "folk theory of democracy" using statistics and facts.

What they refer to as the "folk theory of democracy" is basically what you are taught in school - that democracies are responsive to the will of the people and allow people to shape the policies and laws of the government; that the people decide what the government will do. By analyzing election results and other statistics, they take a number of theories about how democracies allow the people to express their will and test them, and find them all woefully lacking. It turns out that only one theory holds up, and that is that voters reward or punish their representatives based on the voters economic prosperity. But the voters are extremely myopic, only taking into account the few months prior to an election when casting their votes and disregarding the rest of the preceeding couple years.

The Founding Fathers of the US set up a representative democracy because they understood that the normal people wouldn't know enough about politics or policy to really make well-informed decisions. So instead of the people voting on the laws the people would elect representatives that they trust to vote on the laws. The representatives would devote their time to studying and debating the issues and would make well-informed decisions. However the Founding Fathers never anticipated the rise of political parties, which today are so firmly entrenched that most people don't even realize they were never part of the plan. 

The folk theory says that people will choose their party based on their political ideology or policy preferences, but in reality it is just as often the other way - people will develop their policy preference based on their partisan identity. The authors go beyond this to say that the party affiliation is mostly based on a person's "social identity" and has little to nothing to do with their political ideology. The way the book describes it people choose their party affiliation based on the kind of person they consider themselves to be and the kind of people they think belong to the political party. As far as I can tell this is basically a fancy way of saying "peer pressure" - if your family is Republican and your friends are Republican you are likely to be a Republican even if you disagree with Republican policies. In fact, people will often either change their ideology to match their party's, or convince themselves that their party's ideology is closer to their own than it actually is. 

Politics today has become so complex that it is nearly impossible for any normal working person to really understand or make well-informed decisions about all of the policies. In order to be able to handle issues this complex we need to simplify them greatly into mental models which unfortunately omit most of the detail and nuance. Instead of having to consider the myriad sides of an issue and the numerous approaches, we take the talking points that the political parties and the mass media give us and just accept and repeat them. It's a lot easier than having to gather massive amounts of information, sort through it, analyze it and come up with our own opinions. One theory is that political parties provide us with easy cues to figure out what our opinions would be if we had enough time and information for us to come up with them on our own, but this theory is also analyzed and largely debunked.

So if the results of elections have little to do with the policy positions of the candidates and the policy preferences of the voters, then what does drive the elections? Well it turns out it's largely random. Voters will reliably vote out the party in power if the economic wellbeing of the voter has decreased in the months before the election, and vote to keep the party in power if their economic wellbeing has increased just before the election. Voters will also vote out the party in power as a result of things beyond the power of any human to control like floods, droughts, and even shark attacks. But the policy preferences of voters really have little to no effect on elections, other than the fact that many people only develop their policy preferences based on adopting those of the party or candidate they support.

This isn't to say that democracy doesn't work at all, it just doesn't work in the way that it is supposed to work and the way I was taught that it works in school. Because politicians do have to be re-elected they must avoid the appearance of impropriety and appear as if they have the best interests of the people in mind. This at least prevents gross abuses that are typical in dictatorships. But as to whether the people really have much say in determing government policy, it would seem that the answer is no.

Personally I think that the party system in the US is a major factor in this. With only two parties dominating the government, they get their voters worked up about silly issues that aren't really all that important and then once they are in power they are largely indistinguishable, except that they keep their members constantly angry with the other party over these wedge issues which will never be addressed. The only people who really have a say in the government are the wealthy donors and corporations who fund the elections and pay the lobbyists. But that is a different book.

I'm sure this book will upset a lot of people because it challenges some basic assumptions people have about America and about democracy in America. People tend to accept facts that confirm the opinions they already have, and get upset when facts contradict their existing opinions. This book really makes you have to think about democracy and how it works and how it doesn't work. I think this is a book that everyone needs to read.

Etiketten: personal, politics
Keine Kommentare

Laravel Pagination and Ajax

Montag 24 Oktober 2016

I just updated the search of my records here so it would load the results using Ajax instead of refreshing the whole page. Everything seemed to work fine, but then I noticed that it broke the Laravel pagination. I included the pagination in the section of the page reloaded by searches and sorts so it would update appropriately, but then the page links just loaded the results and didn't load the results into the div on the page where they were supposed to be.

This was a bit tricky to solve. What I ended up doing was exporting the pagination views to my resources directory, and then editing it there. To each of the pagination links I added two things:

  • class="page-link"
  • data-val="{ number of page to be loaded}"

Then I added a script to the page that triggers when you click an object with the class of page-link, that gets the page to be loaded from the data-val attribute and submits that to the script that loads the appropriate page with the appropriate variables. Then I did the same thing for the sort links - instead of having each trigger it's own script I made one script triggered by a click on the class and put the data in the data-val.

Etiketten: coding, laravel
Keine Kommentare

I was having a hard time with the Laravel auth package. If you use the out-of-the-box Laravel Auth, if you try to access a page you don't have access to the Auth will redirect you to the login page and then after a successful login redirect you back to the page you were trying to access.

This works fine. But if I go to a page and then click on login it would redirect me back to a page specified in the Auth controller instead of to the page I was on before I clicked login. I searched for a while and found some info, but not much addressing this specific issue. 

I finally found this thread on Laracasts which gives a simple and easy solution to the problem. 

The solution is to override the login form method in the LoginController.php in the app directory. I added this function:

public function showLoginForm(){
     if(!session()->has('url.intended')){
          session()->put('url.intended', url()->previous());
     }
     return view('auth.login');
}

This pushes the previous page onto the session as url.intended, which is the same thing the middleware does. But this does it in all cases, not just when the middleware catches an auth error. After login the Auth controllers now send you back to url.intended instead of to the default page specified in $redirectTo.

Etiketten: coding, laravel
Keine Kommentare

Archiv