Open Data for Open Science

Behind the WorldWide Telescope Theater at SXSW

2013-03-13T19:39:40Z

This past weekend at SXSW, Microsoft and the WorldWide Telescope team worked with NASA, Northrop Grumman and the Space Telescope Science Institute to deliver a one of a kind exhibit around the James Webb Space Telescope (JWST).

The two main anchors of the exhibit was the physical model of the JWST and the WorldWide Telescope Theater. Over the 3 days the WWT Theater hosted over 70 talks using WWT Tours, PowerPoint, and live Skype QnAs – all on a 20 million pixel wall sized display wall.

Behind the scenes – this immersive experience was pulled together by Microsoft Research and our partners for the theater – Epson, NVIDA and Scalable Display Technologies. The 8 Epson projectors worked flawlessly pumping out pixels for over 14 hours a day and this was in a tent without air conditioning and lots of humidity – especially during the first two days of rain. The 8 projectors were driven by 2 NVIDIA Quadro K5000’s in a single PC. The display wall needed the DirectX 11 support to show off the latest WWT Eclipse Alpha build that leverages DX11. Delivering all those pixels to the display wall was one piece, but to make the images seamless and immersive the Scalable Display Manager (SDM) was used. The (SDM) software with it’s EasyBlend technology, made it a snap to wrap and blend the pixels coming from the individual projectors into a really seamless experience.

This whole setup allowed the team to utilize the wall as a very, very large Windows desktop and take advantage of all the Windows applications for the presentations – including

PowerPoint 2013, which takes advantage of DirectX and hardware acceleration and was able to handle the unique aspect ratio so the whole display was used for content
Skype – probably the largest Skype image for Skype session on a desktop you’ve ever seen
OneNote 2013 – used for displaying notes and twitter names for a NASA Social at SXSW
SkyDrive – was used to deliver PowerPoint presentations and WorldWide Telescope Tours to machine driving the display wall and kept them in sync – so edits were always captured
WorldWide Telescope – used to deliver tours exclusive created for the JWST SXSW exhibit.

While most of you might not need to create a wall sized display like this one – it shows how desktop technologies created for Windows scales to create immersive experiences. Let me know if you decide to create one….

More Excel 2013 Goodies - Data Mining Add-ins for Microsoft Office

2013-03-01T00:15:50Z

After posting about Data Explorer yesterday – I thought about other add-ins for Excel that make hard problems easier…and that lead me back to the SQLServerDataMining.com site….it’s great to see that the SQL Server 2012 Data Mining Add-ins for Office (with 64-bit Support) is available for Office 2013.

The Data Mining Add-ins allow you to harness the power of SQL Server 2012 predictive analytics in Excel and Visio and they have been updated to include 64-bit support for Office 2010, and now Office 2013 as well. Use Table Analysis Tools to get insight with a couple of clicks. Or dive into the Data Mining Client for full-lifecycle data mining, and then visualize your models in Visio.

Download Microsoft® SQL Server® 2012 SP1 Data Mining Add-ins for Microsoft® Office® from Official Microsoft Download Center

A tool for Data Science - Microsoft "Data Explorer" Preview for Excel now available

2013-02-28T23:33:08Z

Functionality and features for Excel 2013 just keeps getting better, especially when you realize that Self-Service BI can enable Self-Service SI (Science Intelligence) – the latest is the release of the Microsoft "Data Explorer" Preview for Excel which give you a “intuitive and consistent experience for discovering, combining, and refining data across a wide variety of sources including relational, structured and semi-structured, OData, Web, Hadoop, Azure Marketplace, and more.”

There is a good overview and tutorial available at the Data Explorer help site. I also like all the datasets that can be connected to directly…what other data types/file types would help on the Science side?

PREVIEW NOW AVAILABLE: Microsoft "Data Explorer" Preview for Excel: Try the upcoming capabilities for Self-Service BI with the Microsoft "Data Explorer" Preview for Excel. This preview gives you an early look into upcoming features that simplify access to public and corporate data. Now you can easily discover, combine, and refine data for better analysis in Excel.

Microsoft Data Explorer Preview for Excel

Office Preview | Microsoft BI

AGU 2012–Environment and large displays are fun

2012-12-05T07:01:21Z

This week many of us from the Microsoft Research Earth, Energy and Science team are down at AGU2012 enjoying all that the American Geophysical Union’s 45th annual Fall Meeting has. MSR has been a participant for many years and really enjoy connecting with scientists and researchers to discuss their scientific challenges.

In the Microsoft Research booth this year we have experts from many different areas available to talk about challenges with collecting, visualizing and managing data. We also have Windows 8 and Office 2013 running on the machines to show what is possible from Surface devices, Desktops, and really large touch screens like the 82'” one from Perceptive Pixel in the booth.

You think a tablet is a touch device? Just touch one of these and you’ll realize that pads are mini compared to the 82"

Also we are hosting the following FireSide Chats at the booth tomorrow – Wednesday.

Wednesday 10:00am -10:30 am

Real-Time Water System Management	Prof Barbara Minsker & Jong Sung Lee
	The goal of this project is developing a prototype near-real-time decision support system for river modeling and management in Texas that can serve as a national and international model to promote more sustainable and resilient water systems
Wed 2:00 pm - 2:30 pm
BING: An introduction to Microsoft’s Bing Maps Platform	Erik Lindeman & Ryan Eckardt
	In this session Microsoft will provide an overview of the Bing Maps Platform, its features and how they can be used in a wide range of applications.
Wednesday 3:30pm -4:00pm
Gadgeteer	Steven Johnston
	Microsoft .NET Gadgeteer is an open-source toolkit for building small electronic devices using the .NET Micro Framework and Visual Studio or Visual C# Express.
Wednesday 5:00pm - 5:30pm
DATA MANAGEMENT 101	Carly Strasser
	Tips and tools for managing your research data better

Also glad to see the support for Windows Phone and all HTML apps via the link http://app.core-apps.com/agu-fm12 on that AGU program guide.

Weather forecast on Windows Azure

2012-11-30T22:17:51Z

Wenming Ye is another all-time favorite speakers at our Open Data for Open Science events. His Weather Forecast on Windows Azure demo showcases some of the best features of Windows Azure for big data, and often leaves his audience with eye-opening hands-on experience with Azure.

Recently, at SuperComputing 2012, the Weather Forecast on Windows Azure demo was presented to predict the weather for the next 3 days. As expected, the demo was greatly received by the crowd of computational and weather research professionals. Here is Wenming’s demo script, together with the Live Weather service demo here. This article on Running Weather Research Forecast as a Service on Windows Azure is an excellent reading that can help you get started with reinvigorating classic applications on the cloud service.

If you are interested in Hadoop on Windows Azure, you may find more information here.

Who is not talking about big data today

2012-11-14T05:06:04Z

The São Paulo School of Advanced Science on e-Science for Bioenergy Research, SPSAS e-SciBioenergy, was held Oct. 22-26^th at Brazilian Bioethanol Science and Technology Laboratory – CTBE. I was honored to be one of the invited speakers and had a marvelous time with the students and researchers talking about “big data”.

I learned that the organizing committee selected 79 students from over 250 applications worldwide. My half-day lecture on Open Data for Open Science was the last item on the program, but the room was still full and the discussion was hot. We talked about many ‘big data’ research topics during the lecture. Offline during the breaks, the conversation continued on data and how we can deal with data efficiently by using cutting-edge tools such as WorldWide Telescope, Layerscape, the various Add-on’s for Excel and Word, Windows Azure, and Azure DataMarket. I expect to continue some of the conversations in email.

As one of the students commented “who is not talking about big data today”? the consensus was that “big data” has become such a popular topic because it implies big social impact. Global Warming, biodiversity in the Amazon forest, renewable energy and sugar cane, … these are not just science projects but something that can potentially change our daily life. It attracts talents from multiple scientific domains and computer science to solve the big data problems together. That’s what this event did for the community beyond any given domain and beyond Brazil.

I want to congratulate the Organizing Committee and thank them for a wonderful eScience experience that the event provided for all the participants.

A Game-Changing Mindset in the Field of Sensor Networks

2012-11-13T17:43:04Z

Microsoft Research paper on low energy GPS sensing (called Cloud-Offloaded GPS or CO-GPS) received the Best Paper Award at ACM SenSys’12. Many considered the paper as the best example of the new direction that the community is taking -- from focusing primarily on low power wireless networking to covering broadly sensing, data, and low power devices.

SenSys is the premier conference on networked embedded sensing system and a top forum for sensor network research community. This paper describes a game-changing way to perform location sensing by splitting GPS signal acquisition and processing between embedded devices and the cloud. By logging only a few milliseconds of raw GPS signals, the device can store enough information for resolving GPS-based location with 2~3 orders of magnitude less energy consumption than standalone or mobile phone GPS sensors. The signals are then sent to the cloud with sensor data to reconstruct the location and time that the samples are taken. In delay tolerant data acquisition applications such as animal tracking, float sensor networks, participatory environmental sensing, and long range time synchronization, CO-GPS is the ideal way to extend mobile device battery life. For example, the CO-GPS approach is a key enabling technology in Microsoft Research Project CLEO, a participatory environmental sensing system that we will showcase at the upcoming 2012 AGU Fall Meeting.

Congratulation to Jie and his team!

Reference: Energy Efficient GPS Sensing with Cloud Offloading, Jie Liu, Bodhi Priyantha, Ted Hart, Heitor Ramos, Antonio Loureiro, and Qiang Wang, The 10th ACM Conference on Embedded Networked Sensor Systems (SenSys 2012), Toronto, ON, Canada, Nov, 2012 (http://research.microsoft.com/apps/pubs/default.aspx?id=172624).

StreamInsight Demos

2012-10-20T19:46:00Z

Microsoft StreamInsight™ is a powerful platform that you can use to develop and deploy complex event processing (CEP) applications. Mohamed Ali’s presentation on StreamInsight has been an all-time favorite at our Open Data for Open Science workshops, even when he had to present remotely via Skype over limited internet bandwidth.

His My Campus Walk demo showcases StreamInsight for geospatical stream data analysis together with WorldWide Telescope for spatio-temporal data visualization; the real time geofencing scenario demonstrates simply how StreamInsight supports spatial operators; the Microsoft Shuttle Service demo illustrates how StreamInsight can be used to monitor, analyze and correlate spatio-temporal stream data that is generated by moving objects; and his Intelligent Transportation System demo has been one of the most requested ODOS demos.

Attached is a wonderful article by Mohamed Ali, which describes StreamInsight and the four demos in detail. Please contact e3env@microsoft.com if you have any question or feedback.

Big Data Requires Big Bandwidth - more demos

2012-10-15T18:52:00Z

I mean, big data requires big internet bandwidth to present.

Started in one of our own computer labs on Microsoft campus, since ODOS2011, we have been having great success with our ODOS presentations. Sometimes, I forgot how heavily dependent our live demos are on the internet connection. At the eSceince2012 last week, some of the presenters’ big data live demos didn’t go as smoothly as we hoped for due to limited internet bandwidth.

That reminded me that I should have a backup plan for my future live demos at, for example, the upcoming e-SciBioenergy School. Meanwhile, some of my demo materials can be already shared here, to asynchronously leverage the bandwidth of the internet resources in your offices or research labs.

Here is the information on the Netflix example both Andy Conrad and I referenced on at ODOS-Australia:

Learn OData by the Netflix example from here: http://developer.netflix.com/docs/oData_Catalog
Enabled by the OData API, the Pivot View of NetFlix on the Azure cloud is here: http://netflixpivot.cloudapp.net/

Here is the GeoSearch demo, a faceted geo-search experience enabled by Bing Maps, Windows Azure, and PivotViewer (created by Prof. Phil Yang’s team at George Mason University).

The stunning Puget Sound Flow Model demo, which was demonstrated at the eSceince2012 Adler Planetarium Reception, can be found here. You will need to install WorldWide Telescope (a free download) to see this demo.

If you have any questions, please email us at e3env@microsoft.com.

OData Overview and Sample

2012-10-08T05:02:00Z

Andy Conrad of Microsoft SQL did wonderful presentations on OData at the recent Microsoft Research ODOS workshops. Andy's presentation deck and OData sample code is in the attached. Please email e3env@microsoft.com if you have any comments and/or would like more information.

An Example of the 4th Paradigm in Teaching & Learning Practice

2012-10-05T17:25:43Z

As appraised by many leading astronomical researchers, WorldWide Telescope (WWT) is one of the best implementations of Virtual Observatory (VO), or "big data observatory" nowadays. Meanwhile, we also believe that natural user interface (NUI), such as the Kinect device, can be used to enhance the VO experience. As we are exploring how we may put all these together to benefit our data- and information-intensive research, kids are picking up these technologies naturally in their classrooms. Watch this video and see how WWT and Kinect are making an impact in a middle school in China. The smile on that little girl as she 'navigates in the universe' makes me feel really good about doing my job.

Making a simple WWT tour - The Southern Cross and the Australia Flag

2012-09-26T20:07:00Z

In the WorldWide Telescope, WWT, follow the following steps:

Find and show the Southern Cross constellation

Click Explore, then New, and then Slide-Based Tour…
In the Tour Properties box, enter “Australia Flag” in the Tour Title field and click OK.
Click Add New Slide – a new slide is created
Right-click the new slide and then click Set Start Camera Position
Click View and check Figures and Boundaries
Click Search, enter “crux”, wait for the Crus box to show up and then click the box
Click Australia Flag to get go back to your tour
Right-click on the slide create in step 3, and then click Set End Camera Position
Right-click on the slide again and then click Duplicate Slide at End Position – a new slide is created
On the new slide, click the time label at the bottom and change 10 to 2 (to make the slide duration 2 seconds)
Click the play button to test your new tour
Click Save (on the upper-right of the tour banner)
Enter “Australia Flag” in File Name and click Save
Note that the file extension is wtt

Add the Australia flag

Right-click on the last slide (before the Add New Slide empty slide) and click Duplicate Slide at End Position
At the bottom of the new slide, click the time label and change the duration from 2 to 8 (you may also use the up arrow)
Click Search, enter “crux”, wait for the Crus box to show up and then click the box
Press the “+” sign on your keyboard 5 times to zoom-in
Right-click on the slide and the Set End Camera Position
Click Add New Slide – a new slide is created
Click the Picture icon on the upper right corner
In the pop-up box, enter the file name of a Australia flag image file you have previous saved on your machine, e.g. “Australia.png”
Right click on the image and click Color/Opacity and set Opacity to 50 (so you can see the background in WWT)
Move, resize, and rotate the image so the crux in the flag matches the constellation in WWT
Right click on the image and click Color/Opacity and set Opacity to 10
Right-click on the slide and click Set Start Camera Position
Right click on the flag image and click Color/Opacity and set Opacity to 70
Right-click on the slide and click Set End Camera Position
In the Music box on the upper-right corner, click Browse to fine a music clip you want to use
Click the play button to test your new tour
Click Save

Italian WWT

2012-09-19T18:21:15Z

The Microsoft Research WorldWide Telescope, WWT was introduced to the Italian astronomical research and science education community by Prof. Giuseppe Longo in 2011. Since the WWT workshop at AstroInfomatics2011, Prof. Longo led a series of WWT activities, including 30 some lectures in school on WWT, a nationwide WWT competition, and the Robotic Telescope Ruggero de Ritis Operated through WWT interface offered through Federica Web Learning. What a great use of WWT – it bridges the gaps between astronomical research and science education effectively in a fun way!

20 educational scenarios of AstroInformatics

2012-09-10T04:34:11Z

Inspired by Jim Gray's famous "20 queries" method, the AstroInformatics2012 is set forth to come up with 20 scenarios of astroinformatics for connecting astronomical research with education. I believe the “education” here is for us in computing to learn about the data and computational challenges with Petabytes of astronomical data, for astrophysicists to learn about the latest computing technologies, and for students to learn how astronomical research is conducted today. I’m looking forward to the outcomes on Facebook.

The Aspiration of ODOS

2012-09-10T03:55:52Z

The ODOS here is pronounced oh-dee-oh-es, not oh-dos.

If you search “Open Data for Open Science” on the internet, you will see that this phrase has been a moniker for a series of events Microsoft has produced since early 2011. You will also see that a permanent agenda item for all these events is the Open Data Protocol, OData.

OData is an open Web protocol from Microsoft for querying and updating data. As more and more Microsoft products and technologies are becoming part of the OData ecosystem, we believe that OData can be equally powerful for building solutions to data-intensive science problems. What we really want to do is to use OData to promote and support open science, thus Open Data for Open Science, in short, ODOS.

If you look into our ODOS events on the Web, for example, the ODOS2012 workshop, ODOS Beijing, and ODOS Australia, you will see that they are designed to enable mind-swaps between two distinguished groups of people: experts who create data tools at Microsoft and scientists who need tools to solve their data problems. OData is one of many cutting-edge software technologies we showcase with scientific use scenarios. Do you have a data problem that you may challenge us to create an open solution with you?

Open Data for Open Science 2012, Australia

2012-08-30T04:36:35Z

Learn what the latest technologies from Microsoft can do for data-intensive sciences - join us in Melbourne and Sydney. See http://research.microsoft.com/events/odos2012-australia/.