Recently, Google started flagging up sites which are not mobile friendly in the mobile SERPs by adding a small icon next to these websites. There are now rumours that the search giant might be considering penalising sites that are not mobile friendly.
In a way, it would be beneficial to penalise non-mobile sites only in the mobile search results, and not have the penalty for the main SERPs. However this would bring one major problem, to do this, Google would have to create two different databases of search index: one for mobile and one for non mobile. (Although there are differences in mobile and desktop search related to presentation of extraneous items as highlighted here: http://searchengineland.com/14-differences-between-smartphone-search-desktop-search-results-74687)
And what about those platforms that do not easily fall into this broad category. Is a playstation console considered mobile? What if you do the search with a handheld playstation device? The difficulties of creating cross platform indexes means that such an approach is not feasible.
Therefore, if Google want to maximise the quality of search, they will have to set the penalty on both mobile and non mobile SERPs. So if your site is not mobile ready - then it could be bad news.