tag:blogger.com,1999:blog-325470942024-03-05T01:11:48.776-08:00l'Ishwar incroyable NatarajThis blog is for people to read and give comments to the stuff I write!!!!!!Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.comBlogger85125tag:blogger.com,1999:blog-32547094.post-52245568605085964222019-12-16T01:10:00.001-08:002019-12-17T20:14:49.457-08:00Modern Day Data Warehousing<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCCn-CuVYb2J5yEhNEQ_ZuOMCWSZe5IHZ6SVNNPTVSR85I7cP2UbxYmn19bNwBajlhMIP4qhswSsbbsHlzMR0DKrf7C5adyVuE_k8gQK60-xO68-jNGOmY5Wq4vJcyRRFa6LIl-g/s1600/Testing1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1600" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCCn-CuVYb2J5yEhNEQ_ZuOMCWSZe5IHZ6SVNNPTVSR85I7cP2UbxYmn19bNwBajlhMIP4qhswSsbbsHlzMR0DKrf7C5adyVuE_k8gQK60-xO68-jNGOmY5Wq4vJcyRRFa6LIl-g/s400/Testing1.png" width="400" /></a></div>
<br />
With ever growing data volumes and the need to necessitate a faster approach into making rapid decisions on your data sets, numerous technologies have emerged in the past 5 years trying to blend the transactional pipelines, the data storage and the analytics together. The modern day data warehouse consists of dumping the data into more of a containerized approach catering away from the traditional star and snowflake schemas of the past. Not that these data warehouse schemas have been extinguished but more in tune with rapid assembly and disassembly of data in containers. The core aspect is to whether to de-containerize the data as soon as it comes into our storage layer or go about doing it in an adhoc fashion with your containers being pipelined further into data virtualization stores, data marts or even just reporting on top of the data brought into the data lake, a virtualized data warehouse, an in memory data mart etc...<br />
<br />
Almost all modern day architectures have started moving down this path including Netflix, Facebook that has included its own data storage format on disk from Hive, Microsoft with the CosmosDB predominantly being used by the Bing search team and several others. Real time analytics with modern day queuing systems like Kafka, Rabbit MQ, Kinesis, ZeroMQ etc is taking the analytics world by STORM.... On the virtualized side you have Denodo, Delphix, Dremio etc. which are being used by companies like Virgin Orbit, Chrysler, Biogen, AAA are also making a significant impact in the market nowadays. The general rule of thumb is to ensure that the data virtualization is on top of the Data Warehouse or Data Lake with the key aspect being that this store will be the vault for all the analytics within an organization. Also in virtualized stores, you typically maintain strict copies of the source so as to maintain a total control of the data and improve the data governance of the overall architecture. A data warehouse and a data virtualization layer play different roles that go hand in hand in a BI eco-system. According to wikipedia's definition of a Data Lake -" A data lake is a system or repository of data stored in its natural/raw format, usually object blobs or files. A data lake is usually a single store of all enterprise data including raw copies of source system data and transformed data used for tasks such as reporting, visualization, analytics and machine learning".<br />
<br />
A couple of really good reads on modern day data warehousing are as follows:<br />
<a href="https://cloud.google.com/solutions/build-a-data-lake-on-gcp">https://cloud.google.com/solutions/build-a-data-lake-on-gcp</a><br />
<a href="https://medium.com/netflix-techblog/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977">https://medium.com/netflix-techblog/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977</a><br />
<a href="https://keen.io/blog/architecture-of-giants-data-stacks-at-facebook-netflix-airbnb-and-pinterest/">https://keen.io/blog/architecture-of-giants-data-stacks-at-facebook-netflix-airbnb-and-pinterest/</a><br />
<a href="https://engineering.fb.com/core-data/scaling-the-facebook-data-warehouse-to-300-pb/">https://engineering.fb.com/core-data/scaling-the-facebook-data-warehouse-to-300-pb/</a><br />
<a href="https://customers.microsoft.com/en-gb/story/bing-ads-partner-professional-services-azure-sql-data-warehouse">https://customers.microsoft.com/en-gb/story/bing-ads-partner-professional-services-azure-sql-data-warehouse</a><br />
<a href="https://cloudblogs.microsoft.com/sqlserver/2016/07/12/the-elastic-future-of-data-warehousing/">https://cloudblogs.microsoft.com/sqlserver/2016/07/12/the-elastic-future-of-data-warehousing/</a><br />
<br />
<br />
Note: I am not blogging as much as I want to but will keep posting stuff I find interesting once in a while....<br />
<br />
<br />
<br /></div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-8633903766985371002017-10-23T20:59:00.003-07:002017-10-23T21:00:57.055-07:00TFS Usage <div dir="ltr" style="text-align: left;" trbidi="on">
I was asked a few questions about TFS by a customer a couple of years ago.... I was rummaging through some of my luggage and found the questionairre. Just thought I would answer the questions with this blog post before throwing it away:<br />
<br />
1. How would you breakdown TFS Projects?<br />
The manner in which TFS projects can be broken down is into identifiable work items with tasks associated to them.<br />
<br />
2. Once the Work Item is defined, what are the next steps?<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkhpnFNtU7mGC5dB3Of68Sk2iwQ6PHQgk4htiqPQfikvPrQydyZc8WHvsdJTE0jd10erEvOTf2phgwWADMgJ0QTw3OMFX2atgbJ25enV9c-jMBeV5PEKzCjrDo8bmOmpa4WC2Pew/s1600/TFS1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="184" data-original-width="865" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkhpnFNtU7mGC5dB3Of68Sk2iwQ6PHQgk4htiqPQfikvPrQydyZc8WHvsdJTE0jd10erEvOTf2phgwWADMgJ0QTw3OMFX2atgbJ25enV9c-jMBeV5PEKzCjrDo8bmOmpa4WC2Pew/s640/TFS1.png" width="640" /></a></div>
<h3 style="clear: both; text-align: center;">
The above image does not belong to me but would love to thank the author for the same.</h3>
<br />
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">Review the requirements that
are assigned to the iteration, and define them in more detail.<o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">•Create task work items for
the work that must be performed to implement and test each requirement. Link
the tasks to the requirement work item by using the parent link type.<o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">•Set the Original Estimate
field of each task.<o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">•Divide tasks that have
estimates that are longer than a few days.<o:p></o:p></span></div>
<br />
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">•Compare the estimates with
the time that is available for the iteration. If the estimate total is too
long, simplify some of the requirements, or defer them to later iterations.</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">3. How is a full life cycle implemented in TFS?</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYb7ye_jiYXUk65QmD2G5c4LEfOswQceJ-HEgqB8H64zGbBjsj-xYSJolkbG_sj6_7MW0qihhs1cqshKGz-qcWhYSubA4wVte8B8WXjTpRJ2Xe6UQFKJIi-v4kJS2BTxiD_A65vQ/s1600/TFS2.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1147" data-original-width="1198" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYb7ye_jiYXUk65QmD2G5c4LEfOswQceJ-HEgqB8H64zGbBjsj-xYSJolkbG_sj6_7MW0qihhs1cqshKGz-qcWhYSubA4wVte8B8WXjTpRJ2Xe6UQFKJIi-v4kJS2BTxiD_A65vQ/s400/TFS2.JPG" width="400" /></a></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">4. What are the core definitions for fields and what is the standard naming convention?</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">From my perspective, definition of fields and naming convention is basically up to the organization and its individual representation of how they would like to proceed with the TFS project structure. This can be defined or copied from another template defined by some other organization/person. Would definitely recommend leveraging a template based on the same domain to avoid confusion.</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">Manufacturing --> Manufactureing</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">Pharma --> Pharma</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">Information Technology --> Information Technology e.t.c.</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">The TFS template structure for Scrum as defined by MSDN is as follows (and Microsoft has defined templates for CMMI and Agile as well):</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh633autnguaz6KlRBgE-HMosjm3zZ72o-KNHh90JiGnCDEYJba8ZvvPOKExGVLbCRk6zcdukNnv-WShjUkxL_ObYniOGEIJtUsm3TQaw7KZJX74ci6uyLwzgqsqVgultPO_lFguw/s1600/TFS3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="314" data-original-width="372" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh633autnguaz6KlRBgE-HMosjm3zZ72o-KNHh90JiGnCDEYJba8ZvvPOKExGVLbCRk6zcdukNnv-WShjUkxL_ObYniOGEIJtUsm3TQaw7KZJX74ci6uyLwzgqsqVgultPO_lFguw/s320/TFS3.png" width="320" /></a></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-size: 12pt;">Microsoft's standards are industry standards and would not recommend deviating from the same (or even leverage Google's scrum template as in the following link:</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<a href="https://docs.google.com/spreadsheets/d/1BA9yeFmuA7-mHnvzXLFTLr8LSog_jogoOhs35yLiolA/edit#gid=45">https://docs.google.com/spreadsheets/d/1BA9yeFmuA7-mHnvzXLFTLr8LSog_jogoOhs35yLiolA/edit#gid=45</a></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<br /></div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-36256165570120723812017-09-20T13:53:00.000-07:002017-10-02T05:37:48.761-07:00Branding your SharePoint site in a super fast way - Emgage<div dir="ltr" style="text-align: left;" trbidi="on">
Branding a SharePoint farm on premise or in O365 is an extremely time absorbing task. Recently found a few tools called emgage prime and emgage turbo which basically gives you customized UX branding solutions to make you get to what you want super fast and super easy....Just used it on a SP2016 farm a few days ago and the results were tremendous. A definite recommendation on the SP side of things.</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-48968823104848252842017-05-22T15:04:00.001-07:002019-12-16T01:13:46.259-08:00Load Data into Azure DW using C# in an SSIS script task<div dir="ltr" style="text-align: left;" trbidi="on">
Now there are a lot of reasons why SSIS needs to be leveraged for loading data into the Azure DW platform. Even though Polybase and Azure Data factory are the core criteria's, here are the templates for the SSIS script task that were leveraged to load data (full and incremental into Azure DW) for a specific customer rather than using the data flow task:<br />
<b>SSIS Full Load script</b>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYXDBjELxAXGYmh7fjNyQNl180pheJnUdWaIsfzLvzELxhxamCbXF-_ShRAheubVtRYdk5-5nKAxjLaCJ9AjS_2AHdLbji0AfAQXUJl_O2zh-Q_lHoxHbuAOpG0CiT94kkmdI3LA/s1600/LoadDataSSISScriptTask.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="229" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYXDBjELxAXGYmh7fjNyQNl180pheJnUdWaIsfzLvzELxhxamCbXF-_ShRAheubVtRYdk5-5nKAxjLaCJ9AjS_2AHdLbji0AfAQXUJl_O2zh-Q_lHoxHbuAOpG0CiT94kkmdI3LA/s320/LoadDataSSISScriptTask.PNG" width="320" /></a></div>
<b>SSIS Incremental Load script</b>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDskfscIxPdrkoAZWMbL4qpUfiYxCOgxTafoamVEB8f1jqhxdKclOrJJUhzgwZJe76QNkfe6U3l3sD2aEvUbLllP9_DDKgE7Uz_ua0cUkUOQpQ4hXKl0Ge4qMZoTiFq6cQXQ7qPg/s1600/LoaddataSSIS2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="245" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDskfscIxPdrkoAZWMbL4qpUfiYxCOgxTafoamVEB8f1jqhxdKclOrJJUhzgwZJe76QNkfe6U3l3sD2aEvUbLllP9_DDKgE7Uz_ua0cUkUOQpQ4hXKl0Ge4qMZoTiFq6cQXQ7qPg/s320/LoaddataSSIS2.PNG" width="320" /></a></div>
<br />
<br />
There are a few reasons for this approach and one of them being that the existing package was using a similar structure and one not to be deviated from. The other being that some key logging aspects needed to be handled in a Legacy platform that could not be decommissioned at that time.</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com4tag:blogger.com,1999:blog-32547094.post-28858692984992276532017-05-09T20:45:00.003-07:002017-09-20T13:53:44.410-07:00Why Azure might overtake AWS in its data services offerings<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<ol style="direction: ltr; font-family: Calibri; font-size: 14.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="color: #2e75b5; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 14.0pt; font-style: normal; font-weight: normal;">Compliance
</span></li>
</ol>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Microsoft enterprise
cloud services are independently validated through certifications and
attestations, as well as third-party audits. In-scope services within the
Microsoft Cloud meet key international and industry-specific compliance
standards, such as ISO/IEC 27001 and ISO/IEC 27018, FedRAMP, and SOC 1 and SOC
2. They also meet regional and country-specific standards and contractual
commitments, including the EU Model Clauses, UK G-Cloud, Singapore MTCS, and
Australia CCSL (IRAP). In addition, rigorous third-party audits, such as by the
British Standards Institution and Deloitte, validate the adherence of our cloud
services to the strict requirements these standards mandate.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYnxBnpsNh3p-Nd5VpdEeXg6VZcO7ZTecqTx8WBe8tzIPov4B0pGuij6OrZz1pZw5RdWxEjq1PMxTlBS2oRELHIyZYkAhBoqViGgTWRQEx_lqt0BDzRQnGY2f-cJWlrD740RVfBg/s1600/A1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYnxBnpsNh3p-Nd5VpdEeXg6VZcO7ZTecqTx8WBe8tzIPov4B0pGuij6OrZz1pZw5RdWxEjq1PMxTlBS2oRELHIyZYkAhBoqViGgTWRQEx_lqt0BDzRQnGY2f-cJWlrD740RVfBg/s400/A1.png" width="400" /></a></div>
<br />
<ol style="direction: ltr; font-family: Calibri; font-size: 14.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="color: #2e75b5; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="2"><span style="font-family: "calibri"; font-size: 14.0pt; font-style: normal; font-weight: normal;">Security
</span></li>
</ol>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
With Security
tightly ingrained with its AD offerings, Microsoft currently continues to
evolve its security<span style="mso-spacerun: yes;"> </span>and data integrity
in Azure. Core advantages of Security in Azure are as follows:</div>
<br />
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">Tightly integrated with
Windows Active Directory</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Simplified cloud access using
Single Sign On</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Single and Multi Factor
authentication support</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Rich protocols<span style="mso-spacerun: yes;"> </span>Eg: Federated Authentication
(WS-FEDERATION), SAML, OAuth 2.0 (Version 1.0 still supported), Open ID
Connect, Graph API, Web API 2.0 (in conjunction with
Authentication_JSON_AppService.axd and Authorize attribute)</span></li>
</ol>
<b></b><i></i><u></u><sub></sub><sup></sup><strike><br /></strike>
<br />
<ol style="direction: ltr; font-family: Calibri; font-size: 14.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="color: #2e75b5; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="3"><span style="font-family: "calibri"; font-size: 14.0pt; font-style: normal; font-weight: normal;">Data/BI
Offerings in the Cloud</span></li>
</ol>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Microsoft Azure
relatively has all the required components to support data related and business
intelligence in various formats.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The core database
and data collection/integration formats in Azure are as follows:</div>
<br />
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Data Factory</span></li>
</ul>
<br />
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">Data
Factory is a cloud-based data integration service that orchestrates and
automates the </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">movement</span><span style="font-family: "calibri"; font-size: 11.0pt;"> and </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">transformation</span><span style="font-family: "calibri"; font-size: 11.0pt;"> of data. You can create data integration solutions
using the Data Factory service that can ingest data from various data stores,
transform/process the data, and publish the result data to the data stores.</span><span style="color: #6d7079; font-family: "helvetica neue"; font-size: 8.25pt;">+</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Data Factory service
allows you to create data pipelines that move and transform data, and then run
the pipelines on a specified schedule (hourly, daily, weekly, etc.). It also
provides rich visualizations to display the lineage and dependencies between your
data pipelines, and monitor all your data pipelines from a single unified view
to easily pinpoint issues and setup monitoring alerts.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr4_dN6Et-Ob1AfvXIT1PFXQZhkY12TYjyHzOnaiWZvs0k5M79HgvUkmFSFPHO4xHurSXGaID-pjNfJt1VD5Ee93pJKcTMYlqb1wZFuJxh1MXMNH3AsoEKs9j6YsV7j4Z6hoq1Bg/s1600/A2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr4_dN6Et-Ob1AfvXIT1PFXQZhkY12TYjyHzOnaiWZvs0k5M79HgvUkmFSFPHO4xHurSXGaID-pjNfJt1VD5Ee93pJKcTMYlqb1wZFuJxh1MXMNH3AsoEKs9j6YsV7j4Z6hoq1Bg/s400/A2.png" width="400" /></a></div>
<br />
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">SQL Server
Integration Services (SSIS)</span></li>
</ul>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Leveraging SSIS
which is the core tool leveraged for discontinued development across various
teams, one can move data into and out of Azure to on premise or other cloud
environments based on one's needs. SSIS can integrate databases and data
warehouses in the Azure cloud and also enable individuals to drive templated
based development efforts with ease.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl3-72Vv2bYnOnLCW5KunMqZC87j3e4OuCCH0-4soZNj8BHT_-uTikJ9lKmeK_7M7A5SsUdxoCFv4HO_hS5H9eNUz58Ruu17hzqQiD2g_rEZqORq5EuvK0tREveHDOQjmKuWLJeA/s1600/SSIS+Azzure+-+Page+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl3-72Vv2bYnOnLCW5KunMqZC87j3e4OuCCH0-4soZNj8BHT_-uTikJ9lKmeK_7M7A5SsUdxoCFv4HO_hS5H9eNUz58Ruu17hzqQiD2g_rEZqORq5EuvK0tREveHDOQjmKuWLJeA/s320/SSIS+Azzure+-+Page+1.png" width="320" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Typically compared with AWS data pipeline:</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
AWS Data Pipeline is
a web service that helps you reliably process and move data between different
AWS compute and storage services, as well as on-premise data sources, at
specified intervals. With AWS Data Pipeline, you can regularly access your data
where it’s stored, transform and process it at scale, and efficiently transfer
the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and
Amazon EMR.
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
A pipeline schedules
and runs tasks. You upload your pipeline definition to the pipeline, and then
activate the pipeline. You can edit the pipeline definition for a running
pipeline and activate the pipeline again for it to take effect. You can
deactivate the pipeline, modify a data source, and then activate the pipeline
again. When you are finished with your pipeline, you can delete it. Task Runner polls
for tasks and then performs those tasks. For example, Task Runner could copy
log files to Amazon S3 and launch Amazon EMR clusters. Task Runner is installed
and runs automatically on resources created by your pipeline definitions. You can
write a custom task runner application, or you can use the Task Runner
application that is provided by AWS Data Pipeline. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe8MvOUyKzkdVoahjnrZagwb80t63OhPJq4Pud9FuGGPr-YhhbR8khmLgmlscyhE7lXG2xwbeMRTFtO8XfRfN1O02oGN791W9vPu-hRYu9DSMpuHRtIR08UMvZdcNjNW5PNm6HsQ/s1600/A3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe8MvOUyKzkdVoahjnrZagwb80t63OhPJq4Pud9FuGGPr-YhhbR8khmLgmlscyhE7lXG2xwbeMRTFtO8XfRfN1O02oGN791W9vPu-hRYu9DSMpuHRtIR08UMvZdcNjNW5PNm6HsQ/s320/A3.png" width="320" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Azure SQL
Data Warehouse</span></li>
</ul>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The SQL data
warehouse for Azure maintains storage for data in the realm of files and blobs.
This allows the easy conjunction between dimensional data and measures within
the platform. Core documentation can be found here: <a href="https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-overview-what-is">https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-overview-what-is</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Typically Compared with AWS Redshift:</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Redshift supports
two kinds of sort keys: compound and interleaved. A compound sort key a
combination of multiple columns, one primary column and one or more secondary
columns. A compound sort key helps with joins and where conditions; however,
the performance drops when the query is only on secondary columns without the
primary column. A compound sort key is the default sort type. In interleaved an
sort, each column is given an equal weight. Both compound and interleaved
require a re-index to keep the query performance level high. The architecture
of these systems differ but the end goal is the storage and processing of vast
amounts of data down to second or milli second based result generation.
Focusing on this aspect I am going to give a more detailed insight on Redshift
which is a node based peta byte scaled database as well as a high level
overview of what I recently implemented.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi38BR02USbjANUmYdGiBRhIBRrjvDFEUhhnoe-AUeS-PWcMRnA34UWRl94Sdw37zX9_DiWh7MMlj3piUkVAD0vML4GJQC-bYJKtGDJz8bggpV8YurG7q6f86kLaglFcHcwb5mioQ/s1600/02-NodeRelationships.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi38BR02USbjANUmYdGiBRhIBRrjvDFEUhhnoe-AUeS-PWcMRnA34UWRl94Sdw37zX9_DiWh7MMlj3piUkVAD0vML4GJQC-bYJKtGDJz8bggpV8YurG7q6f86kLaglFcHcwb5mioQ/s320/02-NodeRelationships.png" width="320" /></a></div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">Note:
The above diagram is from the Redshift Warehousing article <span style="background-color: white;">(</span></span><a href="http://docs.aws.amazon.com/redshift/latest/dg/c_high_level_system_architecture.html)"><span style="background: #882222; font-family: "arial"; font-size: 9.9pt;"><span style="background-color: white;">http://docs.aws.amazon.com/redshift/latest/dg/c_high_level_system_architecture.html)</span></span></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If you pay close
attention to the diagram above the compute nodes is responsible for all the
data processing and query transactions on the database and the data nodes
contain the actual node slices. The Leader Node is the service that monitors
the data connections against the Redshift cluster and is also responsible for
the query processing (almost like a syntax checker on the query and functions
leveraged). It then transfers the query across to the Compute Nodes whose main
responsibility is to find the data slices from the data nodes and communicate
with one another to determine the way the transaction needs to be executed. It
is similar to a job broker except that this is more real time than non real
time. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
It is similar to the
analogy of using a bucket..... Consider this:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
You take a bucket
and keep filling water into it, eventually the bucket get filled..... however
what happens when there is an enormous amount of water that needs to be
contained. Either grab a massive bucket or use multiple buckets to store the
water (so the second option actually depicts the Redshift architecture....)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The concept
scaling up implies not only adding a bucket for storage but also a
mechanism to ensure that the pipeline flow goes to an empty bucket which is
nothing but out compute node. But there is the price of the bucket and the cost
of the mechanism that is required to populate the bucket as well....</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="margin-left: .375in; margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">The cons of using Redshift are as follows</span><span style="color: #9c9692; font-family: "arial"; font-size: 9.9pt;">:</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="background: rgb(0 , 255 , 0); color: red; font-family: "arial"; font-size: 9.9pt;">You need an
Administrator to monitor the Redshift cluster at all times</span></li>
<li style="margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="background-color: lime; color: red; font-family: "arial"; font-size: 9.9pt;">The power of
Redshift lies in the manner the database is queried, so a SQL
developer/DBA with understanding of the Redshift internals is definitely a
must</span></li>
<li style="margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="background: rgb(0 , 255 , 0); color: red; font-family: "arial"; font-size: 9.9pt;">Upfront cost
is slightly on the higher side but over a period in time the cost will be
justified with more nodes being added to the cluster</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="color: #9c9692; font-family: Arial; font-size: 9.9pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Redshift
major advantage is the fact that it allows both JDBC and ODBC drivers so definitely if you want to query the
database. The Postgres driver can be found as follows:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="margin-left: .375in; margin: 0in;">
<a href="http://jdbc.postgresql.org/download.html"><span style="font-family: "arial"; font-size: 9.9pt;">http://jdbc.postgresql.org/download.html</span></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="margin-left: .375in; margin: 0in;">
<a href="http://www.postgresql.org/ftp/odbc/versions/msi/"><span style="font-family: "arial"; font-size: 9.9pt;">http://www.postgresql.org/ftp/odbc/versions/msi/</span></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Azure Redis
Cache</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Azure Redis Cache is
based on the popular open-source Redis cache. It gives you access to a secure,
dedicated Redis cache, managed by Microsoft and accessible from any application
within Azure.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Azure Redis Cache is
available in the following tiers:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Basic—Single node,
multiple sizes, ideal for development/test and non-critical workloads. The
basic tier has no SLA.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Standard—A
replicated cache in a two node Primary/Secondary configuration managed by
Microsoft, with a high availability SLA.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Premium—The new
Premium tier includes a high availability SLA and all the Standard-tier
features and more, such as better performance over Basic or Standard-tier
Caches, bigger workloads, disaster recovery, and enhanced security. Additional
features include:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Redis persistence
allows you to persist data stored in Redis cache. You can also take snapshots
and back up the data which you can load in case of a failure.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Redis cluster
automatically shards data across multiple Redis nodes, so you can create
workloads of bigger memory sizes (greater than 53 GB) and get better
performance.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Azure Virtual
Network (VNET) deployment provides enhanced security and isolation for your
Azure Redis Cache, as well as subnets, access control policies, and other
features to further restrict access.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Basic and Standard
caches are availabe in sizes up to 53 GB, and Premium caches are available in
sizes up to 530 GB with more on request.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Azure Redis Cache
helps your application become more responsive even as user load increases. It
leverages the low-latency, high-throughput capabilities of the Redis engine.
This separate, distributed cache layer allows your data tier to scale
independently for more efficient use of compute resources in your application
layer.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Unlike traditional
caches which deal only with key-value pairs, Redis is popular for its highly
performant data types. Redis also supports running atomic operations on these
types, like appending to a string; incrementing the value in a hash; pushing to
a list; computing set intersection, union and difference; or getting the member
with highest ranking in a sorted set. Other features include support for
transactions, pub/sub, Lua scripting, keys with a limited time-to-live, and
configuration settings to make Redis behave more like a traditional cache.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Another key aspect
to Redis success is the healthy, vibrant open source ecosystem built around it.
This is reflected in the diverse set of Redis clients available across multiple
languages. This allows it to be used by nearly any workload you would build
inside of Azure.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Advantages -</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Elasticache Redis:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Simple
query language - no complex features</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Is
(out-of-the-box) not reachable from other regions.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
You're
always limited to the maount of memory</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Sharding
over multiple instances is only possible within your application - Redis
doesn't do anything here</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
You
pay per instance no matter how the load or the number of requests are.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
If
you want redundancy of the data you need to setup replication (not possible
between different regions)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
You
need to work yourself for high availability</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Redis/memcached are
in-memory stores and should generally be faster than DynamoDB </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Exposed as a API</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Typically compared with AWS DynamoDB</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In DynamoDB, data is
partitioned automatically by its hash key. That’s why you will need to choose a
hash key if you’re implementing a GSI. The partitioning logic depends upon two
things: table size and throughput.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
DynamoDB supports
following data types:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Scalar –
Number, String, Binary, Boolean, and Null.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Multi-valued –
String Set, Number Set, and Binary Set.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Document –
List and Map.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Has
a query language which is able to do more complex things (greater than, between
etc.)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Is
reachable via an external internet-facing API (different regions are reachable
without any changes or own infrastructure)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Permissions
based on tables or even rows are possible</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Scales
in terms of data size to infinity</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
You
pay per request -> low request numbers means smaller bill, high request
numbers means higher bill</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Reads
and Writes are different in costs</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Data
is saved redundant by AWS in multiple facilities</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
DynamoDB
is highly available out-of-the-box</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<ul style="text-align: left;">
<li><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Azure Document DB</span></div>
</li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Planet-scale,
highly-available NoSQL database service on Microsoft Azure. Azure DocumentDB is
Microsoft’s multi-tenant distributed database service for managing JSON
documents at Internet scale. DocumentDB is now generally available to Azure
developers. In this paper, we describe the DocumentDB indexing subsystem.
DocumentDB indexing enables automatic indexing of documents without requiring a
schema or secondary indices. Uniquely, DocumentDB provides real-time consistent
queries in the face of very high rates of document updates. As a multi-tenant
service, DocumentDB is designed to operate within extremely frugal resource
budgets while providing predictable performance and robust resource isolation
to its tenants. This paper describes the DocumentDB capabilities, including
document representation, query language, document indexing approach, core index
support, and early production experiences</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">Azure
DocumentDB guarantees less than 10 ms latencies on reads and less than 15 ms
latencies on writes for at least 99% of requests. DocumentDB leverages a
write-optimized, latch-free database engine designed for high-performance
solid-state drives to run in the cloud at global scale. Read and write requests
are always served from your local region while data can be distributed to other
regions across the globe<span style="background-color: lime;"><span style="color: red;">. </span></span></span><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/documentdb-partition-data/"><span style="background-color: lime; color: red; font-family: "segoe ui condensed"; font-size: 11.25pt;">Scale
data</span></a><span style="font-family: "calibri"; font-size: 11.0pt;"> throughput
and storage independently and elastically—not just within one region, but
across many geographically distributed regions. Add capacity to serve millions
of requests per second at a fraction of the cost of other popular NoSQL
databases.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">Easily
build apps at planet scale without the hassle of complex, multiple data center
configurations. Designed as a </span><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/documentdb-distribute-data-globally/"><span style="background-color: lime; color: #cc0000; font-family: "segoe ui condensed"; font-size: 11.25pt;">globally
distributed database system</span></a><span style="font-family: "calibri"; font-size: 11.0pt;">, DocumentDB automatically replicates all of your data to any
number of regions worldwide. Your apps can serve data from a region closest to
your users for fast, uninterrupted access.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">Query
using familiar SQL and JavaScript syntax over document and key-value data
without dealing with schema or secondary indices, ever. Azure DocumentDB is a
truly<span style="background-color: lime; color: red;"> </span></span><a href="http://www.vldb.org/pvldb/vol8/p1668-shukla.pdf"><span style="background-color: lime; color: red; font-family: "segoe ui condensed"; font-size: 11.25pt;">schema-agnostic
database</span></a><span style="font-family: "calibri"; font-size: 11.0pt;"><span style="background-color: lime; color: red;"> </span>capable
of automatically indexing JSON documents. Define your business logic as stored
procedures, triggers, and user-defined functions entirely in JavaScript, and
have them executed directly inside the database engine. Standardized SLA's for
infrastructure throughpout</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Typically compared with AWS DynamoDB</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<span style="font-weight: bold;">Azure HDInsight </span></div>
<br />
<div style="margin-left: .375in; margin: 0in;">
<span style="font-family: "calibri"; font-size: 11.0pt;">Easily spin up enterprise-grade, open source cluster types,
guaranteed with the industry’s best 99.9% SLA and 24/7 support. Our SLA covers
your entire Azure big data solution, not just the virtual machine instances.
HDInsight is architected for full redundancy and high availability, including
head node replication, data geo-replication, and built-in standby NameNode,
making HDInsight resilient to critical failures not addressed in standard
Hadoop implementations. Azure also offers cluster monitoring and 24/7
enterprise support backed by Microsoft and Hortonworks with 37 combined
committers for Hadoop core—more than all other managed cloud providers
combined—ready to support your deployment with the ability to fix and commit
code back to Hadoop.<span style="mso-spacerun: yes;"> </span>Use rich
productivity suites for Hadoop and Spark with your preferred development
environment such as </span><span style="background-color: lime; color: red;"><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/hdinsight-hadoop-visual-studio-tools-get-started/"><span style="font-family: "segoe ui condensed"; font-size: 11.25pt;">Visual
Studio</span></a><span style="font-family: "calibri"; font-size: 11.0pt;">, </span><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/hdinsight-apache-spark-eclipse-tool-plugin/"><span style="font-family: "segoe ui condensed"; font-size: 11.25pt;">Eclipse</span></a><span style="font-family: "calibri"; font-size: 11.0pt;">, and </span><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/hdinsight-apache-spark-intellij-tool-plugin-debug-jobs-remotely/"><span style="font-family: "segoe ui condensed"; font-size: 11.25pt;">IntelliJ</span></a></span><span style="font-family: "calibri"; font-size: 11.0pt;"><span style="background-color: lime; color: red;"> for Scala, Python, R, Java,
and .Net support</span>. Data scientists gain the ability to combine code, statistical
equations, and visualizations to tell a story about their data through
integration with the two most popular notebooks, </span><span style="background-color: lime;"><span style="color: red;"><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/hdinsight-apache-spark-jupyter-notebook-kernels/"><span style="font-family: "segoe ui condensed"; font-size: 11.25pt;">Jupyter</span></a><span style="font-family: "calibri"; font-size: 11.0pt;"> and </span><a href="https://docs.microsoft.com/api/Redirect/en-us/documentation/articles/hdinsight-apache-spark-zeppelin-notebook/"><span style="font-family: "segoe ui condensed"; font-size: 11.25pt;">Zeppelin</span></a></span></span><span style="font-family: "calibri"; font-size: 11.0pt;">. HDInsight is also the only
managed cloud Hadoop solution with integration to Microsoft R Server.
Multi-threaded math libraries and transparent parallelization in R Server
handle up to 1000x more data and up to 50x faster speeds than open source R,
helping you train more accurate models for better predictions than previously
possible. </span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
A
thriving market of independent software vendors (ISVs) provide value-added
solutions across the broader ecosystem of Hadoop. Because every cluster is
extended with edge nodes and script action, HDInsight lets you spin up Hadoop
and Spark clusters that are pre-integrated and pre-tuned with any ISV
application out-of-the-box, including Datameer, Cask, AtScale, and StreamSets.)</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Integration
with the Azure Load Balancer:</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Azure
Load Balancer delivers high availability and network performance to your
applications. It is a Layer 4 (TCP, UDP) load balancer that distributes
incoming traffic among healthy instances of services defined in a load-balanced
set.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Azure
Load Balancer can be configured to:</div>
<br />
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .75in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-bottom: 0pt; margin-top: 0; margin-top: 3pt; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Load
balance incoming Internet traffic to virtual machines. This configuration
is known as </span><a href="https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-internet-overview"><span style="font-family: "calibri"; font-size: 11.0pt;">Internet-facing load
balancing</span></a><span style="font-family: "calibri"; font-size: 11.0pt;">.</span></li>
<li style="margin-bottom: 0; margin-bottom: 0pt; margin-top: 0; margin-top: 3pt; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Load
balance traffic between virtual machines in a virtual network, between
virtual machines in cloud services, or between on-premises computers and
virtual machines in a cross-premises virtual network. This configuration
is known as </span><a href="https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-internal-overview"><span style="font-family: "calibri"; font-size: 11.0pt;">internal load balancing</span></a><span style="font-family: "calibri"; font-size: 11.0pt;">.</span></li>
<li style="margin-bottom: 0; margin-bottom: 0pt; margin-top: 0; margin-top: 3pt; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Forward
external traffic to a specific virtual machine.</span></li>
</ul>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
All
resources in the cloud need a public IP address to be reachable from the
Internet. The cloud infrastructure in Azure uses non-routable IP addresses for
its resources. Azure uses network address translation (NAT) with public IP
addresses to communicate to the Internet.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
*not
similar to the elastic emr load balancer</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<a href="https://www.trustradius.com/products/microsoft-azure-hdinsight/reviews/pros-and-cons">Microsoft
Azure HDInsight Pros</a></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .75in; margin: 0in;">
Shows
live changes in analytics.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .75in; margin: 0in;">
Azure
in itself is very user-friendly, HDInsight is a great addition. </div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .75in; margin: 0in;">
Works
in tandem with other Microsoft technologies like Power BI and Excel seamlessly
as end clients</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<a href="https://www.trustradius.com/products/microsoft-azure-hdinsight/reviews/pros-and-cons">Microsoft
Azure HDInsight Cons</a></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
When
loading large volumes of data, issues in data might stop the load mid way due
to either data corruption errors issues in data might stop the load mid way due
to either data corruption errors or some latency issue. The entire load needs
to be tranisitioned from start.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Azure HDInsight is a service
that provisions Apache Hadoop in the Azure cloud, providing a software
framework designed to manage, analyze and report on big data.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">HDInsight clusters are
configured to store data directly in Azure Blob storage, which provides
low latency and increased elasticity in performance and cost choices.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Unlike the first edition of
HDInsight , now it is delivered on Linux – as Hadoop should be, which
means access to to HDP features. The cluster can be accessed via Ambari in
the web browser, or directly via SSH.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">HDInsight has always been an
elastic platform for data processing. In today’s platform, it’s even more
scalable. Not only can nodes be added and removed from a running cluster,
but individual node size can be controlled which means the cluster can be
highly optimized to run the specific jobs that are scheduled.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">In its initial form, there
were many options for developing HDInsight processing jobs. Today,
however, there are really great options available that enable developers
to build data processing applications in whatever environment they prefer.
For Windows developers, HDInsight has a rich plugin for Visual Studio that
supports the creation of Hive, Pig, and Storm applications. For Linux or
Windows developers, HDInsight has plugins for both IntelliJ IDEA and
Eclipse, two very popular open-source Java IDE platforms. HDInsight also
supports PowerShell, Bash, and Windows command inputs to allow for
scripting of job workflows.</span></li>
</ul>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<span style="font-weight: bold;">Typically Compared with AWS EMR</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Amazon
EMR provides a managed Hadoop framework that makes it easy, fast, and
cost-effective to process vast amounts of data across dynamically scalable
Amazon EC2 instances. You can also run other popular distributed frameworks
such as <a href="https://aws.amazon.com/emr/details/spark/">Apache Spark</a>, <a href="https://aws.amazon.com/emr/details/hbase/">HBase</a>, <a href="https://aws.amazon.com/emr/details/presto/">Presto,</a> and <a href="https://aws.amazon.com/blogs/big-data/use-apache-flink-on-amazon-emr/">Flink</a> in
Amazon EMR, and interact with data in other AWS data stores such as Amazon S3
and Amazon DynamoDB.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Amazon
EMR securely and reliably handles a broad set of big data use cases, including
log analysis, web indexing, data transformations (ETL), machine learning,
financial analysis, scientific simulation, and bioinformatics.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
AWS
EMR is more mature when compared to HDInsight however HDInsight development
continues to progress at a rapid pace compared to the phased approach for AWS
EMR.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
AWS
EMR pricing is also higher than Azure HDInsight in terms of storage usage.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
EMR
release velocity is better than Azure HDInsight.</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<br />
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Amazon EMR provides a managed
Hadoop framework that simplifies big data processing.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Other popular distributed
frameworks such as Apache Spark and Presto can also be run in Amazon EMR.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Pricing of Amazon EMR is
simple and predictable: Payment can be done on hourly rate. A 10-node
Hadoop can be launched for as little as $0.15 per hour. Because Amazon EMR
has native support for Amazon EC2 Spot and Reserved Instances, 50-80% can
also be saved on the cost of the underlying instances.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">It also is in vogue due to
its easy usage capability. When a cluster is launched on Amazon EMR the
web service allocates the virtual server instances and configures them
with the needed software for you. Within minutes you can have a cluster
configured and ready to run your Hadoop application.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">It is resizable, the number
of virtual clusters depending on the processing needs can be easily
contracted or expanded.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Amazon EMR integrates with
popular business intelligence (BI) tools such as Tableau, MicroStrategy,
and Datameer. For more information, see Use Business Intelligence Tools
with Amazon EMR.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">You can run Amazon EMR in a
Amazon VPC in which you configure networking and security rules. Amazon
EMR also supports IAM users and roles which you can use to control access
to your cluster and permissions that restrict what others can do on the
cluster. For more information, see Configure Access to the Cluster.</span></li>
</ul>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike></div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-69824017601457212562016-11-30T17:58:00.000-08:002016-12-01T13:22:06.163-08:00Modern Day Messaging Patterns<div dir="ltr" style="text-align: left;" trbidi="on">
Its been a few days now and I have been focused on understanding modern day messaging patterns for a problem I am trying to solve. I do know that there are existing server side tools like Active MQ, Rabbit MQ and even WMS that can do the trick and already have pre-defined patterns tested and validated for performance and security but in this case even though I am not trying to reinvent the wheel in terms of creating a new pattern or any of these server side products, I am definitely trying to understand the manner in which these products have been created and if I can actually leverage some of the principles in a server side application I am writing up. For example in modern based web application development, if .Net based, you have patterns like the one's defined here: <a href="https://msdn.microsoft.com/en-us/library/aa480027.aspx">Microsoft SOA patterns</a> that do the neat tricks you would need. Man at times I feel I am going at 300 miles an hour without any crash guards: Code reviews, Custom product development, Customer Engagements, Team management, generating pipelines for my practices, reviewing and helping the team in analyzing problems, Big Data Workloads, Machine Learning, my data science degree etc... have consumed majority of my life for the past 2.5 years. Thank goodness I am going to be taking an extended break early next year for my brother's wedding. Here are the gist of things I am trying to solve though:<br />
<br />
1. Dynamic reformatting of my messages<br />
2. Integration with Machine Learning to collaborate with specific scientic models (pre created)<br />
3. Templated messaging with dynamic parameterization<br />
4. Distribution channel modification<br />
5. AI Hub and Spoke messaging relay<br />
<br />
<br />
<br /></div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-11855346803810565372016-10-19T15:55:00.002-07:002017-09-20T13:56:14.048-07:00Power BI To Embed Or Not To Embed<div dir="ltr" style="text-align: left;" trbidi="on">
It is very critical for organizations to work & play with data. Power BI - the reporting solution from Microsoft is literally scorching the market with its rapid pace in usage. On a quick note while interacting with your Power BI report like the following--><br />
<iframe allowfullscreen="true" frameborder="0" height="600" src="https://app.powerbi.com/view?r=eyJrIjoiYjhjNzMzZTktMzFhYi00OGY3LWE0M2YtYmFmMzExZWU2MDU1IiwidCI6IjY1ODM2MzZiLWQxNTYtNDkyZS04NmNlLWI4ZmNjYjc5MGRmMSIsImMiOjN9" width="500"></iframe><br />
This report is accessible by the public. In order to create a more personalized/advanced security reporting structure with Power BI, the Power BI embedded would be the way to go. Create a workspace collection in Azure and then generate the required API keys (two by default - primary and secondary). These API keys will be leveraged by your web application. Once this is done create the pbix solution file in your desktop tool and publish or import the pbix solution to the Azure workspace using powershell/C#/ruby/java etc... Now to interact with the pbix file in your application, you need to leverage the Power BI Embed API's. However there is another approach using Power BI API's instead of the embed API's. The embed API's is a pay as you go service where you are using the Page views determining the pricing and the other option. The manner in which an embed Power BI report acts vs a
straight forward Power BI report is that you are registering the Power BI
workspace in the case of an embedded report .<br />
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<o:p> </o:p>But in a straight forward Power BI report, you will be publishing
the web application in which the Power BI report is consumed using the Power BI API’s
to Azure. The core difference is that there will be more development
required here vs the previous method.</div>
<div class="MsoNormal">
____________________________________FIN________________________________________</div>
<div class="MsoNormal">
<o:p></o:p></div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com2tag:blogger.com,1999:blog-32547094.post-25761552754700749182016-06-15T14:57:00.001-07:002016-06-15T15:05:20.567-07:00Microsoft acquires LinkedIn<div dir="ltr" style="text-align: left;" trbidi="on">
On Monday 6/13/2016, Microsoft announced its acquisition of LinkedIn. This is a major game changer in the world of IT. But before we get to some of the advantages of this acquisition, Microsoft actually was working on a LinkedIn killer on its CRM dynamics platform. The idea was to generate more footprint for its CRM solution as well as create something unique with it. This was started in early 2012 and was way before its actual acquisition of LinkedIn. Here are my thoughts into where this acquisition will lead Microsoft & LinkedIn to:<br />
<br />
<br />
<ul style="text-align: left;">
<li>Microsoft gains a huge database of professionals and organizations in various streams: This alone is the most massive gain by Microsoft. It could start targeting professionals/organizations to either move onto the Microsoft platform or join the Microsoft platform which can bolster its sales by a huge margin/</li>
<li>Microsoft integration of LinkedIn ads with Bing: Just imagine an organization trying to establish a marketing campaign. Now with LinkedIn ads and Bing ads integrated, an organization will have more opportunities to get page views or clicks and the potential to accelerate CTR's to conversions. This could come with a potential increase in cost of a campaign but it might make a really profitable decision.</li>
<li>Changes in technology trends at Microsoft will accelerate: Now since the foundation of LinkedIn is based on Cloud computing and Big Data, LinkedIn would have significantly made a lot of strides in terms of architectural and open source technology aspects. Now if these can be converted to products on the Azure suite, Microsoft might make a sizable profit on it. Also this will increase the trend of embracing open source vs closed technology stacks. (Azkaban, Voldemort, Increased usage of Kafka and Rabbit MQ etc...)</li>
<li>Added stream of Revenue: During an interview with Satya and Jeff, Satya did mention the integration with O365 and Azure as the major driver and Jeff mentioned that it made sense for LinkedIn to sell at this point. But I think the potential driver for this deal was the significant monetary gain that Microsoft will have in terms of Ads and revenue generated from LinkedIn subscribers. LinkedIn had major competition coming its way in the form of more local or regional based corporate social websites, but with this it actually gives it a major edge over its competitors due to Microsoft's global reach. Azure and O365 could potentially outlets for Microsoft to create LinkedIn based apps similar to Yammer</li>
<li>Allow organizations to create internal social platforms using the technology gained by LinkedIn: Microsoft could potentially make a configurable LinkedIn app for all its devices inclusive of the XBOX that organizations can tap into and create internal social networking platforms.</li>
</ul>
<div>
<br /></div>
<div>
It would be fun to just do a prediction as to where this acquisition would lead Microsoft. probably a story to tell another day.</div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-89807353800761410492016-05-24T20:18:00.001-07:002016-05-24T20:19:54.873-07:00R - Notes<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The following are basically my notes while studying R and is meant as a reference point for myself <half and="" course="" doing="" for="" have="" i="" make="" might="" nbsp="" no="" others="" p="" sense="" the="" things="" was="" when="" work="" written=""></half><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Just a few pointers to anyone preparing for R or studying R:</div>
<div style="margin: 0in;">
</div>
<ul style="text-align: left;">
<li style="font-family: Calibri; font-size: 11pt;"><span style="font-size: 11pt;">Take a quick look at your statistical math basics before proceeding</span></li>
<li style="font-family: Calibri; font-size: 11pt;"><span style="font-size: 11pt;">Before applying any formula on your base data, try to understand what the formula is and how it was derived (this will make it easier for one to understand)</span></li>
<li style="font-family: Calibri; font-size: 11pt;"><span style="font-size: 11pt;">Use it in tangent with the Data Analysis in Excel</span></li>
<li><span style="font-family: "calibri"; font-size: 11pt;">Refer to the cheat sheets available on </span><span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><a href="https://www.rstudio.com/resources/cheatsheets/">https://www.rstudio.com/resources/cheatsheets/</a></span></span></li>
<li><span style="font-family: "calibri";">Segregate the workbench for each module</span></li>
<li><span style="font-family: "calibri";">There are best practices that can be incorporated while programming in R</span></li>
<li><span style="font-family: "calibri";">Try and jot notes when and where one can... </span></li>
<li><span style="font-family: "calibri";">Refer to existing data-sets embedded in R before jumping into a data.gov file</span></li>
<li><span style="font-family: "calibri";">Refer to R programs written already in Azure ML <it an="" and="" are="" gives="" kind="" of="" these="" understanding="" used...="" when="" why="" you=""></it></span></li>
</ul>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">rnorm() by default
has mean 0 and variance 1</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
head() has its own
built in precision </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">*default settings in
R can be modified by the options() function</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
example:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
options(digits = 15)
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
#will display 15
digits (Max digit for option display --> 22 and min digit --> 0): Error
if > 22 --> Error in options(digits = 30) : </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
#invalid 'digits'
parameter, allowed 0...22</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
#Infinity Operations
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Inf/0 --> Inf</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Inf * 0 --> Inf</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Inf + 0 + (0/0)
--> NaN</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Inf + 0 --> Inf</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*The ls() lists all
the variable stored in R memory at a given point in time</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*rm() will remove
contents from the list</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*To figure out the
commands in R use the following command ? followed by the function that needs
to be leveraged:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
?c()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
?rand()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
?max()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Functions and
Datastructures</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sin()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
integrate()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
plot()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
paste()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Again single valued
functions and multi valued functions</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*A special vector is
called a factor</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
gl() --> generate
levels</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*creating a function
in R</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
test<-function p="" x=""></-function><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
{</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
x=5</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
return (x*x+(x^2))</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
}</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*for loop in R</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
l*apply() vs
sapply()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Binding elements</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
rbind() --> bind
elements in a matrix in a row manner</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
cbind() --> bind
elements in a matrix in a columnar manner</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Every vector/matrix
has a data mode....</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
logical</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
numerical</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Can be found using
mode()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*dimensions in
matrices</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
=defines the number
of rows and columns in a matrix</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*can be used with
dimnames(),rownames(),columnnames()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*Navigating through
R package libraries really bad....</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*HMISC -->
Harrell misc... Contains many functions useful for data analysis, high-level
graphics, utility operations, functions for computing sample size and power,
importing and annotating datasets, imputing missing values, advanced table
making, variable clustering, character string manipulation, conversion of R
objects to LaTeX code, and recoding variables.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*R search path is
the R working directory</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
getwd() --> get
working directory</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
setwd()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*to read in a table
format:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
testfile <- filename="" p="" read.table=""></-><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
read.csv</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
read.csv2</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
read.fwf (fixed
width file)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*readLines()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
scan()--> reads a
content of a file into a list or vector</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
f*ile() connections
can create connections to files for read/write purposes</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
write.table(line,file="myfile",append=TRUE)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
f1<-file p=""></-file><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
close(f1)-->
close the file connection</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
write.table(dataFieldName,filename)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
write.csv</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
write.csv2</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
base::sink Send
R Output to a File</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
dump()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
dput() --> save
complicated R objects (in ASCII format)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
dget() -->
inverse of dput()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*file in conjunction
with open="w" option</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
R has its own
internal binary object</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
use save() &
load() for binary format</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*RODBC Package</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Common Functions</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
odbcDriverConnect(Connection)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlQuery()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlTable()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlFetch()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlColumns()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
close(Connection)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*specify the version
of the driver TDS_Version=8.0 and which port to use default:1433.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Ex: </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlFetch(conn,"Tablename")</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
query<- from="" p="" selet="" t1="" t2="" test=""></-><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlQuery(conn,Query)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlColumns(conn,"Tablename")</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
sqlColumns(conn,"Tablename")[c("COLUMN_NAME"),c("TYPE_NAME")]</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
check dimensions of
a table using dim()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
*summary() ->
gives a range of stats on the underlying vector,list,matrix</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhucQU3dXu-rdGXu77-62VJj_p58_LpfpGAHBi48fEhgREKw0Qn1mfEt_qxy1_7eY9bVmI18DOFKE7QntdTmO7VnI1fBmNHAL87dcl7yHb9JBW7HmNZwBY-H6J3bSTgQkXhaTtcjw/s1600/R1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhucQU3dXu-rdGXu77-62VJj_p58_LpfpGAHBi48fEhgREKw0Qn1mfEt_qxy1_7eY9bVmI18DOFKE7QntdTmO7VnI1fBmNHAL87dcl7yHb9JBW7HmNZwBY-H6J3bSTgQkXhaTtcjw/s1600/R1.png" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Which function
should you use to display the structure of an R object?</div>
<div style="color: #595959; font-family: Calibri; font-size: 9.0pt; margin: 0in;">
Str()</div>
<div style="color: #595959; font-family: Calibri; font-size: 9.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Log(dataframe) to
investigate the data </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Calculate Groups</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
tapply()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
aggregate()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
by()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRk1B4y5uDkb27D5iPD_viWcz7FdDpBSmBBpCVEzUG2cKJMfRAneAGGzK2s0TD6o3OEaVXALgMEEhyTSFE0UWaXhOWJ32wVjdY4oTcxnc_jW541FPty97QavbKToW0ugBNrn1lZQ/s1600/R2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRk1B4y5uDkb27D5iPD_viWcz7FdDpBSmBBpCVEzUG2cKJMfRAneAGGzK2s0TD6o3OEaVXALgMEEhyTSFE0UWaXhOWJ32wVjdY4oTcxnc_jW541FPty97QavbKToW0ugBNrn1lZQ/s1600/R2.png" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Attach()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Detach()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Convert to frequency
using prop.table()</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Simulations in R </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
MCMC (Markov Chain
Monte Carlo)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Encryption</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Performance Testing</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Drawback -->
Uncertainity</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Pseudo Random Number
Generator - The Mersenne Twister</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Mersenne Prime</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
set.seed(number)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
rnorm(3)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Uniform distribution
- runif(5,min=1,max=2)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Normal distribution
- rnorm(5,mean=2,sd=1)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Gamma distribution -
rgamma(5,shape=2,rate=1)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Binomial
distribution -rbinom(5,size=100,prob=.3)</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Multinomial
Distribution - rmultinom(5,size=100,prob=c(.2,.4,.7))</div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">Regression:</span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">eruption.lm = lm(eruptions ~ waiting, data=faithful)</span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">coeffs = coefficients(eruption.lm)</span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">coeffs</span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">coeffs[1]</span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">coeffs[2]</span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">waiting = 80 # the waiting time </span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">duration = coeffs[1] + coeffs[2]*waiting </span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;">duration --> Predicted value</span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
loadd ggplot2 or ggplot using load("gplot")</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Compare models using ANOVA</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: black;"><span style="color: white;">X1 <- nbsp="" span="" style="font-family: 'Lucida Console', 'courier new', monospace; font-size: 13px; line-height: 19.5px;">lm(y ~ x1 + x2 + x3 + x4, data=mydata)</-></span></span></div>
</div>
</div>
</div>
</div>
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: black; color: white;"><span style="font-family: "lucida console" , "courier new" , monospace; font-size: 13px; line-height: 19.5px;">Y1 <- lm="" span="" x1="" x2="" y=""><br style="font-family: 'Lucida Console', 'courier new', monospace; font-size: 13px; line-height: 19.5px; margin: 0px; padding: 0px;" /><span style="font-family: "lucida console" , "courier new" , monospace; font-size: 13px; line-height: 19.5px;">anova(X1, Y1)</span></-></span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: black; color: white;"><span style="font-family: "lucida console" , "courier new" , monospace; font-size: 13px; line-height: 19.5px;"><br /></span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: black; color: white;"><span style="font-family: "lucida console" , "courier new" , monospace; font-size: 13px; line-height: 19.5px;"><br /></span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: black; color: white;"><span style="font-family: "lucida console" , "courier new" , monospace; font-size: 13px; line-height: 19.5px;"><br /></span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: black; color: white;"><span style="font-family: "lucida console" , "courier new" , monospace; font-size: 13px; line-height: 19.5px;"><br /></span></span></div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-79654891409254196512016-02-13T14:55:00.001-08:002016-02-13T16:15:04.432-08:00Hadoop Installation on Win 10 OS<div dir="ltr" style="text-align: left;" trbidi="on">
Setting the Hadoop files prior to Spark installation on Win 10:<br />
1. Ensure that your JAVA_HOME is properly set. A recommended approach here is to navigate to the installed Java folder in Program Files and copy the contents into a new folder<br />
you can locate easily for eg:- <b>C:\Projects\Java</b>.<br />
2. Create a user variable called JAVA_HOME <note: an="" delete="" entry="" had="" i="" in="" issue="" java="" oracle="" path="" so="" the="" this="" to="" used="" variable="" with=""> and enter "<b>C:\Projects\Java</b>"</note:><br />
3. Add to the path system variable the following entry: "<b>C:\Projects\Java\Bin;</b>"<br />
4. Create a HADOOP_HOME variable and specify the root path that contains all the Hadoop files for eg:- "<b>C:\Projects\Hadoop</b>"<br />
5. Add to the path variable the bin location for your Hadoop repository: "<b>C:\Projects\Hadoop\bin</b>" <Keep <note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version="">track of your Hadoop installs like C:\Projects\Hadoop\2_5_0\bin><br />
6. Once these variables are set, open command prompt as an administrator and run the following commands to ensure that everything is set correctly:<br />
A] java<br />
B] javac<br />
C] Hadoop<br />
D] Hadoop Version <if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><br />
7. Also ensure your winutils.exe is in the Hadoop bin location.<br />
<either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with="">< Download the same from - <a href="https://www.barik.net/archive/2015/01/19/172716/">https://www.barik.net/archive/2015/01/19/172716/</a>><br />
8. Also an error might related to the onfiguration location might occur -Add the following to the hadoop-env.cmd file to rectify the issue:<br /><b>
set HADOOP_IDENT_STRING=%USERNAME%<br />
set HADOOP_PREFIX=C:\Projects\Hadoop<br />
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop<br />
set YARN_CONF_DIR=%HADOOP_CONF_DIR%<br />
set PATH=%PATH%;%HADOOP_PREFIX%\bin</b><br />
9. Another issue that I did face while leveraging Hadoop 2.6.0 install was the issue with the hadoop.dll. I had to recompile the source using MS VS to generate the hadoop.dll and pdb files and replaced the hadoop.dll which came along with the install.<br />
10. Another error that I faced was "The system cannot find the batch label specified - nodemanager". Replace all the "<b>\n</b>" characters in the Yarn.cmd file to "<b>\r\n</b>".<br />
11. Also replace the "<b>\n</b>" characters in the Hadoop.cmd file to "<b>\r\n</b>".</either></if></note:><br />
<note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version=""><if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with="">12. Yarn-site.xml change is as shown in the screenshot below:</either></if></note:><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3PjUQIG3tmN1vKMCsygY50LA0cDu5HQgkJSwr6_YBzGgvR4xOwXkLEVFVVlqJEifatALNJDAC_VjLIlU56sAwQE1OIEHdqKInqjte7273DK3x44aWPyqbVoV2wBxH_ZJvc71I0g/s1600/Yarn.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="211" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3PjUQIG3tmN1vKMCsygY50LA0cDu5HQgkJSwr6_YBzGgvR4xOwXkLEVFVVlqJEifatALNJDAC_VjLIlU56sAwQE1OIEHdqKInqjte7273DK3x44aWPyqbVoV2wBxH_ZJvc71I0g/s320/Yarn.PNG" width="320" /></a></div>
<note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version=""><if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with=""><br />13. Make changes to the core-site.xml as shown in the screenshot below:
</either></if></note:><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAMoiXh3dvAlV0jZzZC-_qyBJxnvXMrhBzrhCeDKSeiIMJX4d4XuyCd75IGTM5aIW7leWyiM10mq4x5IdviC6eMH817NxK2aBLApUzaSYnff0k8TrPJHxg_qFYjvi00qMBYR-BVQ/s1600/core-site.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAMoiXh3dvAlV0jZzZC-_qyBJxnvXMrhBzrhCeDKSeiIMJX4d4XuyCd75IGTM5aIW7leWyiM10mq4x5IdviC6eMH817NxK2aBLApUzaSYnff0k8TrPJHxg_qFYjvi00qMBYR-BVQ/s320/core-site.PNG" width="320" /></a></div>
<note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version=""><if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with=""><br /></either></if></note:>
<note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version=""><if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with="">14. Make the configuration changes as per the answer here :</either></if></note:><br />
<note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version=""><if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with=""><a href="http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows/23959201#23959201">http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows/23959201#23959201</a></either></if></note:><br />
<note: according="" be="" best="" can="" for="" keeping="" p="" practices="" specified="" the="" to="" version=""><if and="" another="" code="" correctly="" errors="" file="" hadoop-env.cmd="" hard="" here="" is="" it="" java_home="" method="" modify="" not="" p="" path.="" set="" the="" through="" to="" will="" your=""><either and="" apache="" comes="" compile="" download="" exe:="" executable="" following="" generate="" hadoop="" it="" leverage="" link="" long="" nbsp="" or="" p="" source="" standard="" that="" the="" to="" with="">15. Download Eclipse Helios for your Win OS to generate the jar's required for your map reduce applications. Use jdk1.7.0_71 and not the 1.8+ versions to compile your hadoop mapreduce programs.<br />16. Kickstart your Hadoop dfs and yarn and add data from any of your data sources and get ready to map reduce the heck out of it.... < A quick note,after formatting your named node it defaults to a tmp folder along with your machine name... in my case it is C:\tmp\hadoop-myPC\dfs\data><br />
</either></if></note:></div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-52994452361393470282015-12-21T17:08:00.001-08:002015-12-21T17:23:05.965-08:00Tableau Dashboards Published...<div dir="ltr" style="text-align: left;" trbidi="on">
A few Tableau dashboards I have published off late to give a flair for different visualizations within Tableau:<br />
<script src="https://public.tableau.com/javascripts/api/viz_v1.js" type="text/javascript"></script><br />
<div class="tableauPlaceholder" style="height: 500px; width: 600px;">
<noscript><a href='#'><img alt='Prescriber Sales ' src='https://public.tableau.com/static/images/Pr/PrescriberSales/PrescriberSales/1_rss.png' style='border: none' /></a></noscript><object class="tableauViz" height="500" style="display: none;" width="600"><param name='host_url' value='https%3A%2F%2Fpublic.tableau.com%2F' /> <param name='site_root' value='' /><param name='name' value='PrescriberSales/PrescriberSales' /><param name='tabs' value='no' /><param name='toolbar' value='yes' /><param name='static_image' value='https://public.tableau.com/static/images/Pr/PrescriberSales/PrescriberSales/1.png' /> <param name='animate_transition' value='yes' /><param name='display_static_image' value='yes' /><param name='display_spinner' value='yes' /><param name='display_overlay' value='yes' /><param name='display_count' value='yes' /><param name='showVizHome' value='no' /><param name='showTabs' value='y' /><param name='bootstrapWhenNotified' value='true' /></object></div>
<br />
<br />
<br />
<br />
<br />
<br />
<div class="tableauPlaceholder" style="height: 500px; width: 600px;">
<noscript><a href='#'><img alt=' ' src='https://public.tableau.com/static/images/Ad/AdvertisingSample/AdvertisingAnalysis/1_rss.png' style='border: none' /></a></noscript><object class="tableauViz" height="500" style="display: none;" width="600"><param name='host_url' value='https%3A%2F%2Fpublic.tableau.com%2F' /> <param name='site_root' value='' /><param name='name' value='AdvertisingSample/AdvertisingAnalysis' /><param name='tabs' value='yes' /><param name='toolbar' value='yes' /><param name='static_image' value='https://public.tableau.com/static/images/Ad/AdvertisingSample/AdvertisingAnalysis/1.png' /> <param name='animate_transition' value='yes' /><param name='display_static_image' value='yes' /><param name='display_spinner' value='yes' /><param name='display_overlay' value='yes' /><param name='display_count' value='yes' /><param name='showVizHome' value='no' /><param name='showTabs' value='y' /><param name='bootstrapWhenNotified' value='true' /></object></div>
</div>Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-37540143359812197642015-11-22T02:02:00.003-08:002018-04-21T13:23:53.898-07:00Cyclotron's Android App<div dir="ltr" style="text-align: left;" trbidi="on">
Just created a Xamarin Android mobile application <stand alone="" for="" now="">. Extremely easy to use and did not require much reading the resources to understand how to go about building it. The first iteration is as shown in the figure below:</stand><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEir-y1cwagNN4RK6VoNJq8qQLSdwmAt5oC0K_x2OlCM0oCYmgM9nkC3aYeYEy4rW-zXIhusBa-1Q3ECgx5jIbWpsZSBhFO61Qe9UjMICUdLSIG3DINSlf8iglTAbxeTV7f07UwfXw/s1600/CycMobileApp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEir-y1cwagNN4RK6VoNJq8qQLSdwmAt5oC0K_x2OlCM0oCYmgM9nkC3aYeYEy4rW-zXIhusBa-1Q3ECgx5jIbWpsZSBhFO61Qe9UjMICUdLSIG3DINSlf8iglTAbxeTV7f07UwfXw/s400/CycMobileApp.png" width="400" /></a></div>
Though the emulator (Nexus 5 through 7) did not render as clearly I wanted it to, but still a great start for v1.0. The next version would basically integrate with Google maps. As soon as one clicks the app, you get a splash screen and then navigate to Cyclotron's main menu from where you can navigate to the layouts. The support aspect would also be a part of the next iteration of the app along with the Login. There were a few more images added to the individual activities <rendered layout="" on="" one="">.</rendered><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaWZ3PB5mXzUJ0h2Eph89ciAb0x6V_YF5LX265bx0k89p80OYt6jMXFDyGHlmoS_LABhAu1Ew8qQUNxled9eL1e20YngFPGAyiao5_Lz_Tl5T7I_rdhHwPE9N8vIHy3Izh8xLofg/s1600/MobPic2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaWZ3PB5mXzUJ0h2Eph89ciAb0x6V_YF5LX265bx0k89p80OYt6jMXFDyGHlmoS_LABhAu1Ew8qQUNxled9eL1e20YngFPGAyiao5_Lz_Tl5T7I_rdhHwPE9N8vIHy3Izh8xLofg/s320/MobPic2.PNG" width="196" /></a></div>
<div>
Probably leverage this article as the initial help on how to use the app.... The follow up items are as follows:</div>
<div>
1. Integration with Google Maps</div>
<div>
2. Synchronization with Cyclotron's support database</div>
<div>
3. Login for Support</div>
<div>
4. Tweak the UI<br />
5. Replicate for IOS</div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-23215257743985254242015-10-20T22:18:00.000-07:002015-10-22T20:36:16.755-07:00Excel Regular Expression Parsing<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Sample piece of code to parse Excel files with regular expressions using Excel Query (The first piece does not loop through all the rows to make it more efficient and fetches the resultant set based on the first row. The second piece goes through all the columns and finds a match):<br />
<br />
<h3 style="text-align: left;">
<span style="background-color: white;"><b>//This piece uses ExcelQuery --></b></span></h3>
<b style="background-color: white; color: #741b47;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System;</span></b></div>
<div style="background-color: white;">
<div class="MsoNormal">
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Collections.Generic;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Linq;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Web;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.IO;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Text;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Web.UI;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Text.RegularExpressions;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Web.UI.WebControls;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
Scanning;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
System.Data;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">using</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
Innovative.Data;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">//Install ExcelQuery</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">namespace</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
WebApplication1</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">public</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">partial</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">class</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">WebForm1</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> :
System.Web.UI.</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Page</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> {</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">protected</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">void</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> Page_Load(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">object</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
sender, </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">EventArgs</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> e)</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ExcelQuery</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> excelQuery = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">new</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ExcelQuery</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"C:\\Test\\Test.xlsx"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
excelQuery.HeaderHasFieldNames = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">true</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
excelQuery.Provider = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ExcelQuery</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ConnectionStringItems</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Providers</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.Jet12;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
excelQuery.ExcelVersion = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ExcelQuery</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ConnectionStringItems</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ExcelVersions</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.Excel2007to2013;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
excelQuery.ConnectionProperties = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"IMEX=0"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">bool</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> test = searchResults(excelQuery);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
Response.Write(test.ToString());</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> }</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">protected</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">bool</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> searchResults(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">ExcelQuery</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
excelQuery)</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">var</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> x = excelQuery.GetSheets();</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">foreach</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">string</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> y </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">in</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> x)</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">int</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> i = 1,a=1;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">string</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> sql = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"SELECT * FROM
["</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> + y+</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"]"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Regex</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> numberRegex = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">new</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Regex</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">@"\d{3}-\d{2}-\d{4}"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">, </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">RegexOptions</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.IgnoreCase);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">DataSet</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> data = excelQuery.ExecuteDataSet(sql,</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"Table"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">while</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (i <= 1)</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">foreach</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">DataTable</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> dt </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">in</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
data.Tables)</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">do</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
a++;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">var</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> rowAsString = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">string</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.Join(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">",
"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">, dt.Rows[0].ItemArray);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Match</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
match = numberRegex.Match(rowAsString);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">if</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (match.Success)</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
{</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">return</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">true</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
}</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
} </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">while</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (a <= 1);</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> i++;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
}</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
}</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
}</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">return</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">false</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">;</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
}</span><o:p></o:p></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: #741b47;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> }</span><o:p></o:p></b></span></div>
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="color: #741b47;"><b>} </b></span> </span><br />
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><br /></span>
<br />
<h3 style="text-align: left;">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: purple; font-family: Consolas; font-size: 9.5pt;"><b>//This piece uses ClosedXML and loops through columns</b></span></h3>
<br />
<b style="color: purple;"> Regex numberRegex = new Regex(@"\d{3}-\d{2}-\d{4}", RegexOptions.IgnoreCase);</b><br />
<span style="color: purple;"><b> XLWorkbook wbk = new XLWorkbook("C:\\Test.xlsx");</b></span><br />
<span style="color: purple;"><b> //List<string> results;</string></b></span><br />
<span style="color: purple;"><b><br /></b></span>
<span style="color: purple;"><b> foreach (IXLWorksheet worksheet in wbk.Worksheets)</b></span><br />
<span style="color: purple;"><b> {</b></span><br />
<span style="color: purple;"><b> //var test = worksheet.ToString();</b></span><br />
<span style="color: purple;"><b> var companyRange = worksheet.RangeUsed();</b></span><br />
<span style="color: purple;"><b> var companyTable = companyRange.AsTable();</b></span><br />
<span style="color: purple;"><b><br /></b></span>
<span style="color: purple;"><b> if (companyTable != null)</b></span><br />
<span style="color: purple;"><b> {</b></span><br />
<span style="color: purple;"><b> </b></span><b style="color: purple;">var results = companyTable.DataRange.Rows()</b><br />
<span style="color: purple;"><b> .Where(companyRow => numberRegex.IsMatch(companyRow.Field(x).GetString()) == true)</b></span><br />
<span style="color: purple;"><b> .ToList();</b></span><br />
<span style="color: purple;"><b><br /></b></span>
<span style="color: purple;"><b> int cnt = results.Count;</b></span><br />
<span style="color: purple;"><b><br /></b></span>
<span style="color: purple;"><b> if (cnt > 0)</b></span><br />
<span style="color: purple;"><b> {</b></span><br />
<span style="color: purple;"><b> Response.Write("Testing");</b></span><br />
<span style="color: purple;"><b> }</b></span><br />
<span style="color: purple;"><b><br /></b></span>
<span style="color: purple;"><b> }</b></span><br />
<span style="color: purple;"><b> }</b></span><br />
<br />
<h3 style="text-align: left;">
<b><span style="color: purple;">//This Piece just loops through Cells Used</span></b></h3>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">byte</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">[] array = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">File</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.ReadAllBytes(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"C:\\Projects\\Sheet1test.xlsx"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">);<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Stream</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> stream
= </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">new</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">MemoryStream</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">(array);<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Regex</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
numberRegex = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">new</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">Regex</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">@"\d{3}-\d{2}-\d{4}"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">, </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">RegexOptions</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">.IgnoreCase);<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">XLWorkbook</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
wbk = </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">new</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">XLWorkbook</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">(stream);<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">foreach</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">IXLWorksheet</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
worksheet </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">in</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
wbk.Worksheets)<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="color: purple;"><b>
{<o:p></o:p></b></span></span></div>
<div class="MsoNormal">
<b style="color: purple;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> </span></b></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">IXLCells</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
cells = worksheet.Columns().CellsUsed();<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">var</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> results =
cells.Where(cellval => numberRegex.IsMatch(cellval.GetString()) == </span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">true</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">).ToList();<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">int</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> cnt =
results.Count;<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">if</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"> (cnt > 0)<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="color: purple;"><b>
{<o:p></o:p></b></span></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
Response.Write(</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">"Testing"</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">);<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="color: purple;"><b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">
</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">return</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></b></span></div>
<div class="MsoNormal">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="color: purple;"><b>
}<o:p></o:p></b></span></span></div>
<div class="MsoNormal">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="color: purple;"><b> }<o:p></o:p></b></span></span></div>
<div class="MsoNormal">
<b><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="color: purple;"> </span></span></b></div>
<br />
<div class="MsoNormal">
<br /></div>
</div>
</div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-24492479896465849492015-09-07T20:16:00.003-07:002015-09-16T15:58:27.970-07:00Elasticsearch Notes <div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Been recently playing with a lot of open source tool sets to figure out core solutions for different product ideas that I have. One of the recent technologies I have used is the Elasticsearch tool. Elasticsearch is basically a NoSql based indexing solution that allows one to use Lucene indexes on top of massive data sets especially string based documents. This blog post is just a bunch of notes that I have compiled. What is Elasticsearch?<br />
Elasticsearch is a document store with each document stored as an index in a cluster with multiple shards. Sharding is basically a concept of partitioning data based on some metric within the data:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9WjkAW_O0Hmj8-PUmDx7pPpNEcFrf-ZI9LkP9dIjI-MQy4mgVbOmFVaqi6Mep5BMLQEVszNKFXDVxZEjfPumVdJSlmYblNHRXzXQy3JO6hHh9_d6St29pxtTbfZ75Ozhp9U12Dg/s1600/elasticsearch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="61" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9WjkAW_O0Hmj8-PUmDx7pPpNEcFrf-ZI9LkP9dIjI-MQy4mgVbOmFVaqi6Mep5BMLQEVszNKFXDVxZEjfPumVdJSlmYblNHRXzXQy3JO6hHh9_d6St29pxtTbfZ75Ozhp9U12Dg/s200/elasticsearch.png" width="200" /></a></div>
Now Elasticsearch exposes an http based request-response to query the individual documents stored in the index.<br />
In my case I created a 2 node cluster as shown in the following image:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh20ZO9aHqMRANxXXS0DqvJBSZRsyCa3Z3KNJU3P9GMjsThr48NFYZBXb8767tInHsKAtjV_Xdpxp9HCiga4HkwKkk9UrB-kttogH3uzIOyImmdVsox5RFfGFzKcAa56X1AdNO6Vw/s1600/elasticsearch02.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="116" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh20ZO9aHqMRANxXXS0DqvJBSZRsyCa3Z3KNJU3P9GMjsThr48NFYZBXb8767tInHsKAtjV_Xdpxp9HCiga4HkwKkk9UrB-kttogH3uzIOyImmdVsox5RFfGFzKcAa56X1AdNO6Vw/s320/elasticsearch02.PNG" width="320" /></a></div>
<br />
After this step I created an index called imdb_search <just a="" few="" i="" in="" movies="" past="" recent="" that="" the="" to="" validate="" wanted="" watched="">. Initially wanted to create a Graphing tool to showcase the connections that I had in facebook and the relations between them...but then I decided nah....will go with a more open public api search for JSON feeds. Note: I used the SENSE api provided by Marvel for the GET/POST/TRACE http commands to store/parse/write the data. </just><br />
<br />
<br />
<table border="1">
<tbody>
<tr>
<td font="bold">POST /imdb_test<br />
{<br />
"mapping":{<br />
"post": {<br />
"_routing":{<br />
"required":"true",<br />
"path":"movie_name"<br />
},<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"properties":{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> "movie_id":{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>type:"int"<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> },<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"movie_name":{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>type:"string"<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> },<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"movie_box_office_value":{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>type: "integer"<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> },<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"movie_date_of_release":{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>type: "date"<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> }<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> }<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}<br />
}</td>
</tr>
</tbody></table>
<br />
A note here is that the nodes can be reconfigured with a API call:<br />
<span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">PUT </span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">/</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">_cluster</span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">/</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">settings
</span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">{</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">
</span><span class="str" style="border: 0px; box-sizing: border-box; color: #008800; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">"persistent"</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;"> </span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">:</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;"> </span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">{</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">
</span><span class="str" style="border: 0px; box-sizing: border-box; color: #008800; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">"discovery.zen.minimum_master_nodes"</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;"> </span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">:</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;"> </span><span class="lit" style="border: 0px; box-sizing: border-box; color: #006666; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">2</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">
</span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">}</span><span class="pln" style="border: 0px; box-sizing: border-box; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">
</span><span class="pun" style="border: 0px; box-sizing: border-box; color: #666600; font-family: Consolas, Menlo, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console'; font-size: 16.2px; font-stretch: inherit; line-height: 24.3px; margin: 0px; padding: 0px; vertical-align: baseline; white-space: pre;">}</span><br />
In this case I created an index with the following mappings --> movie_id, movie_name, movie_box_office_value, movie_date_of_release with the data types as shown in the above table.<br />
<br />
Next I pulled the required JSON feeds for the documents from the imdb open api's. A couple of the queries I used are as follows:<br />
<a href="http://www.imdb.com/xml/find?json=1&nr=1&nm=on&q=Disney" style="background-color: white; color: #1155cc; font-family: arial, sans-serif; font-size: 12.8px;" target="_blank">http://www.imdb.com/xml/find?<wbr></wbr>json=1&nr=1&nm=on&q=Disney</a><br />
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<a href="http://www.imdb.com/xml/find?json=1&nr=1&nm=on&q=Marvel" style="color: #1155cc;" target="_blank">http://www.imdb.com/xml/find?<wbr></wbr>json=1&nr=1&nm=on&q=Marvel</a></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<a href="http://www.imdb.com/xml/find?json=1&nr=1&nm=on&q=Lucas" style="color: #1155cc;" target="_blank">http://www.imdb.com/xml/find?<wbr></wbr>json=1&nr=1&nm=on&q=Lucas</a></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
Using a mix and match of some of the data and generating random box office and date results in my temp C# parser, I dynamically created a few entries for my Elasticsearch document repository. A couple of the POSTS are as follows:</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">
<br /></div>
<br /></div>
<table border="1">
<tbody>
<tr>
<td>POST imdb_test/post
{
"movie_name":"Star Wars: Episode VI - Return of the Jedi",
"movie_id":1,
"movie_box_office_value":7000000,
"movie_date_of_release":"2005-07-01"
}
</td>
</tr>
<tr>
<td>POST imdb_test/post
{
"movie_name":"Terminator",
"movie_id":2,
"movie_box_office_value":10000000,
"movie_date_of_release":"1994-06-04"
}
</td>
</tr>
</tbody></table>
<br />
<div>
<br /></div>
<div>
After generating a couple of data files for my index.... I was able to query the list with specific filters/ basic queries using _search like GET /imdb_test/_search or more complex queries like</div>
<div>
<div>
GET imdb_test/post/_search</div>
<div>
{</div>
<div>
"query": {</div>
<div>
"match": {</div>
<div>
"user_name": "terminator"</div>
<div>
}</div>
<div>
}</div>
<div>
,</div>
<div>
"aggs": {</div>
<div>
"all_words": {</div>
<div>
"terms": {"field":"movie_name"}</div>
<div>
}</div>
<div>
}</div>
<div>
}</div>
</div>
<div>
I just stepped short after checking out my routing using the GET commands:</div>
<div>
GET /imdb_test/post/Terminator</div>
<div>
The next step was to create a SPA to generate a dashboard with the resultant set. The advantages of using Elasticsearch is primary its querying abilities on massive volume sets and can be useful in terms of document repositories like in my case, Blogging and even Geo based analysis of data. It works with JSON documents and also has a really cool analytics dashboard to show case the metrics of the environment called Kibana. We can also alias our index like in my example we can categorize the movies into Horror/Action/Comedy etc. genres by aliasing which is a pretty handy feature.<br />
<br /></div>
<div>
[9/16/2015]<br />
On that note, I took the above ElasticSearch setttings and applied a Life Sciences need for the same. I stored my meta data results for the graphs inside Elastic Search and pulled in the required metric for the same. Now in my case I needed the graphs to be jazzy --> so I had 3 indexes sets with two seperate aliases each. And I changed the data feeds to the same. For the genetic algorithm, I basically leveraged the json feed from Karsten @ http://www.karstenahnert.com/. Also got a good idea for the dashboard from Colin @http://colinwhite.net/Dash2.5/ which was more for Hospital management.<br />
<br />
Here is a sneek peek at the dashboard:-<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguPR-muEeSJL-ztn72EcdeolQHM7MaQNsUb9NShUuV82O1BLcAV6YaJRvZ0Y_RnT1w2YfS20XVFMHxH_Q1b53VwFQKKKg_8ZUX4gcJ7Y-KTNGf3Uwng-vmOK7TNj8XImH3IyZYKw/s1600/20150916-154403_capture.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguPR-muEeSJL-ztn72EcdeolQHM7MaQNsUb9NShUuV82O1BLcAV6YaJRvZ0Y_RnT1w2YfS20XVFMHxH_Q1b53VwFQKKKg_8ZUX4gcJ7Y-KTNGf3Uwng-vmOK7TNj8XImH3IyZYKw/s1600/20150916-154403_capture.gif" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<note: --="" --blue="" -="" a="" an="" and="" are="" data="" different="" for="" green....="" green="" heat="" hot="" i.e.="" just="" luke="" map="" mimic="" nascent="" rect="" red="" squares="" svg="" take="" that="" the="" to="" topics="" trends="" try="" using="" warm="" was="" yellow=""></note:><br />
Note: The red, green and blue were to mimic a heat map with trends that are hot - Red, luke warm - green and shallow - yellow. Used simple SVG rect for that portion.<br />
The visualizations were done using D3 and the gets were fetched using Angular.....<br />
<br /></div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-83067332919326303992015-01-26T15:23:00.002-08:002015-01-27T16:43:23.349-08:00Data Implosion Continued....<div dir="ltr" style="text-align: left;" trbidi="on">
Now this post is a part 2 to my previous post on Data Implosion and not Explosion. Off late I have been reading a lot of books on data especially related to Data Science and one of the books I am on the verge of completion is Developing Analytic Talent: Becoming a Data Scientist by Vincent Granville. Now the reason I purchased this book was to get a deeper insight into what Data Scienists do and what is required in order to become one. The unfortunate part of this book is that getting a deeper insight into Data Science is absolutely incomprehensible. The reason behind this is that majority of the formulae (which I would love to understand and apply) are not defined clearly. And majority of these formulae's do not have a definition of the co-efficients used. Coming from a Computer Science background, I am familiar with the core aspects like the Big O notation and how to derive it from one's code, so reading through these parts was eating with a spoon since I do leverage these concepts on a regular basis (or whenever I actively code). However the statistical representation of data barring frequency tables ( a few other aspects like mean, average and standard deviation) are understandable. When it comes to certain formulae like weights and sample spaces are not clearly represented and are garbled. But I would definitly recommend the chapter on Excel which I think has been written fantabulously and is a must read for data geeks. When it comes to clustering and when to leverage certain principles like Sampling for example again was very shaky for me to read and understand. However I am going to talk about cluster sampling in this article:<br />
1. Let us consider a population of 500 people.<br />
2. Divide this sample space into clusters: For example-><br />
A] number of men (140)<br />
B] number of women (150)<br />
C] number of children (210)<br />
3. Now each cluster is a sample in its own right and we will not combine them together but delineate this after step 2.<br />
4. Now in a random population sample, we choose a subset of men, women and children from our respective clusters and apply the necessary parameters we need to gauge, a couple of examples here would be:<br />
A] What time does one sleep?<br />
B] What time does one have lunch?<br />
C] How many times in a month does one shop? etc..<br />
<br />
5. Let us take only [A] for our current problem.<br />
--> I choose 20 from each subset as a sample<br />
Let us consider Men-<br />
Space Definition Number of Men who sleep Average<br />
--------------------------------------------------------------------------------------------------------------------------<br />
1pm - 5pm 4 4/20 =0.20<br />
6pm - 10pm 12 12/20 =3/5 =0.60<br />
post 10 pm 4 4/20 =0.20<br />
--------------------------------------------------------------------------------------------------------------------------<br />
20 1<br />
<br />
From this we can apply the same to the overall Men's cluster and gather the required metrics.<br />
<br />
Now clustering sampling can actually bombard your analysis if your data points are redundant and is well not worth the time and effort. But as I mentioned earlier a Data Implosion Mechanic is someone who will not deduce these aspects of the data volume but rather force the metrics that need to be infringed on the data. I could be wrong here but this is my take on it... Let's move onto Monte Carlo simulations a bit....<br />
Let us consider [B] and we allocate 2 time slots 12pm-1pm and 1pm-2pm and let us say that none of the people who took the survey were willing to respond to this specific question.<br />
So let us randomize a 12pm - 1 pm(1) and 1pm - 2pm (0) for the 150 women:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIvhUVJqPsnwWU6ucWy2n4wb9FpV-ePVwJ8p9YrjACpjZGyOxdi4xu7Lxe95doGjvCigT99vbjkk_iaQYGJCz4F6XnylGICKaAxbcPiTLXQN0KF35fM6RLx_d372p1yqB7zj35zw/s1600/MCSimulation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIvhUVJqPsnwWU6ucWy2n4wb9FpV-ePVwJ8p9YrjACpjZGyOxdi4xu7Lxe95doGjvCigT99vbjkk_iaQYGJCz4F6XnylGICKaAxbcPiTLXQN0KF35fM6RLx_d372p1yqB7zj35zw/s1600/MCSimulation.png" height="320" width="140" /></a></div>
In this case let us take an average of the 150 entries we have 44% of the women take their meal between 12 pm - 1 pm and the remaining 56% take their meal after 1pm. So if I were to set up a Sandwich shop, I might consider going after the 1pm slot to open the shop..... (Since this is more of random application of metrics, you could cross reverse the data results and then try your experiment).<br />
<table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse; width: 64px;">
<colgroup><col style="width: 48pt;" width="64"></col>
<tbody>
<tr height="20" style="height: 15pt;">
<td align="right" height="20" style="background-color: transparent; border: 0px black; height: 15pt; width: 48pt;" width="64"><span style="font-family: Calibri;"></span></td>
</tr>
</tbody></colgroup></table>
<br />
<br />
<br />
<br />
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-37596173362166305252014-10-22T17:56:00.002-07:002014-10-28T01:32:09.836-07:00Data Implosion and not Explosion<div dir="ltr" style="text-align: left;" trbidi="on">
I was planning on writing this article a while ago but never came down to it.... but I do have some time on my hands to write about how data has changed over the years. Nowadays data has been imploding and what I want to infer from implosion is that the manner in which data is being collated and published to end users with various transformations along the way creates a sense of cause and effect. You have massive data-sets but the information from this can be co-related in so many different ways that eventually one is unable to figure out how to go about getting the required information onto the end user's field of vision. Let us consider Big Data solutions....now whenever we feel that the volume of data has reached its significant end in terms of storage limitations we can go about introducing a solution (a big data solution) in order to contain the explosion and ensure that there is no data loss.... hence in this case we can always ensure that in the eventuality of a massive uptake in terms of data we can always control the size limitations by doing something concrete. But now comes the important aspect of how multiple streams of well contained data actually bump into each other to try and make information ambiguous to end users. Let us consider a database which contains the list of movies released in the past 5 years..... This is a massive amount of data but we still have containers in place to collect it. Now lets say we need to figure out which were successful (not only in terms of monetary aspects) and which were not. Initially this sounds simple and we think that performing this endeavor is going to be easy but as we look into the data, there are so many metrics which need to be considered here:<br />
<br />
A] Box Office Revenue per region<br />
B] Number of Theaters released in per region<br />
C] Actors in the movie<br />
D] Production Budget<br />
E] Director of the Movie<br />
F] Studio responsible for the movie<br />
G] Genre of the movie<br />
H] Region based dissemination of people who watch the particular genre<br />
I] Video Rentals and Digital sales<br />
<br />
And the list goes on.... even though all these data points are present it is still difficult to actually decipher the data and figure out which movie can be considered a success (box office, critically acclaimed, fan acclaimed etc...) or not because the data being stored is actually imploding at such a massive scale that it becomes in-deterministic to even collate meaningful representation of data from the container.<br />
Example here:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxCQKCgE-wt-WWyQwYa70AY7K822A3_d72sV_rhyphenhyphen82yeWYAjqnsRQjL8ikuz9kxHJrjTNjl1Nz4B392lj4R8VZfV0BfTrIQfm0g2jpe3pBegn99p5288vVWeQyo_u7htD3gzeT8Q/s1600/ImgX.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxCQKCgE-wt-WWyQwYa70AY7K822A3_d72sV_rhyphenhyphen82yeWYAjqnsRQjL8ikuz9kxHJrjTNjl1Nz4B392lj4R8VZfV0BfTrIQfm0g2jpe3pBegn99p5288vVWeQyo_u7htD3gzeT8Q/s1600/ImgX.png" height="247" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The above figure showcases how the metrics are gathered and the manner in which they collide</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4AdL6KQ-zK1qAB4e2dXEYXd4FQ2tycn4nNfnCTpPmcKQwqf-q2uXj_4Wdb7uNixP3W07u1W5Mfx_tN7uWG-YclQDzKw18s_6_Y3Z1hyksqM6Z7Uj7ApGcSzO37kz6ByzA0ZDL5w/s1600/ImgY.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4AdL6KQ-zK1qAB4e2dXEYXd4FQ2tycn4nNfnCTpPmcKQwqf-q2uXj_4Wdb7uNixP3W07u1W5Mfx_tN7uWG-YclQDzKw18s_6_Y3Z1hyksqM6Z7Uj7ApGcSzO37kz6ByzA0ZDL5w/s1600/ImgY.png" height="159" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Even Creating hierarchies becomes an incredible effort</td></tr>
</tbody></table>
<br />
<br />
Now the important aspect over here is to pre-determine what scenario one can gain from the massive volume of data and create a standard or norm which will be the end all for all metrics guidelines. This eventually will have to be a criteria decided among the masses and not an individual entity which comes back to my earlier statement that Data is imploding at a massive rate and not exploding. We need to introduce implosion mechanisms and maybe that will eventually be a future job by itself. Think about it--> Data Implosion Handler @ xyz co..... interesting yet is eventually bound to happen.</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-63004871162958809702014-08-19T23:47:00.001-07:002014-08-20T00:59:47.834-07:00Blue Screen of Death --> Microsoft Power Map<div dir="ltr" style="text-align: left;" trbidi="on">
I was fiddling around with Power Map and wanted to see how far I could go ahead with the mapping visualization.......<br />
I basically leveraged the Power Station data file available at:<br />
<a href="http://office.microsoft.com/en-us/excel-help/redir/XT104048055.aspx?CTT=5&origin=HA104091224">http://office.microsoft.com/en-us/excel-help/redir/XT104048055.aspx?CTT=5&origin=HA104091224</a><br />
<br />
After downloading this excel file, I started to create Power Map with the basic idea of having 3 layers.<br />
1. The first layer would take into account all the counties based on the Power Transmitted<br />
2. The second layer would consist of all the companies based the Power distribution<br />
3. The third layer considers the power distribution based on Plant Name....<br />
<br />
Now this excel file contains roughly about 20,000 rows...... My machine configuration is basically a Windows 8 with an Intel i7 core and 16 GB of RAM. I was good up to step 1 in the steps described above but when I hit step 2 that's when I noticed that my memory just bloated like crazy...... I decided like a crazy person to try step 3 and that's when my machine crashed. Unfortunately I could not take screen shots as well as would not like to reproduce this issue. But to just give a general idea even playing around with roughly 20,000 rows of data and using the above layering with probably close to 8 GB of free space on my box killed my machine with even the disk usage going to 100%. I decided to work on a smaller subset and show you what exactly I was trying to produce (This is for the state of California with a much lesser subset of counties):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6MW3EWVmCW8HlKdOw-D-Lb3rEZY9_OgeLTUUkN0vmKj2NqubDWVkaODf7_kEqRR79yes5jh-fke6hucAhvPHeGKYV0iCXYzAOvNokr_qLXjZIG6AqHtcAig2wAxJoCdRhkTmxTA/s1600/PMAP1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6MW3EWVmCW8HlKdOw-D-Lb3rEZY9_OgeLTUUkN0vmKj2NqubDWVkaODf7_kEqRR79yes5jh-fke6hucAhvPHeGKYV0iCXYzAOvNokr_qLXjZIG6AqHtcAig2wAxJoCdRhkTmxTA/s1600/PMAP1.png" height="229" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Step 1 is shown in the above diagram<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqeqARvBwKunrjwv4gDZVLiZv57fK70X6jSTemdhqmeAjDmO3XLAXU2VreRz3RyBzlLUPTwv7N3bwMH3954GWu87z-NlTULYXbalRw2cD7rzSsy3RQOYe3VF2h8aXanCZyWfCs5Q/s1600/PMAP2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqeqARvBwKunrjwv4gDZVLiZv57fK70X6jSTemdhqmeAjDmO3XLAXU2VreRz3RyBzlLUPTwv7N3bwMH3954GWu87z-NlTULYXbalRw2cD7rzSsy3RQOYe3VF2h8aXanCZyWfCs5Q/s1600/PMAP2.png" height="222" width="320" /></a></div>
<br />
Step 2 is as shown in the above diagram<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeZvadiHNkYQg_2KnN2BUyRc4GNZ1tvEkLP4ryEoqxYU_OqK19BQapiWwiGgmX_XrQjRsLXVxU3aM70oldGYyk8lK4hmkiZx5VIi1hA1cYQhI09fT0LgNegTrc74lwHhZt2lDpXQ/s1600/PMAP3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeZvadiHNkYQg_2KnN2BUyRc4GNZ1tvEkLP4ryEoqxYU_OqK19BQapiWwiGgmX_XrQjRsLXVxU3aM70oldGYyk8lK4hmkiZx5VIi1hA1cYQhI09fT0LgNegTrc74lwHhZt2lDpXQ/s1600/PMAP3.png" height="202" width="320" /></a></div>
<br />
The above figure is very close to what I wanted to do but I used a clustered column instead. I would definitely recommend going slow while creating a Power Map. It does provide mind blowing visualization but it is dependent on the amount of memory one can leverage at any given point in time and can blow up if the resultant data set is way too large. The above examples were created using about 500 rows of data. I was way more cautious as I did not want to blow my machine again. But at the end of the day, the map was beyond imagination in terms of visualization of data points. Post a comment on this post if this has happened to you as well!!</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-39224275284872936352014-08-11T12:03:00.002-07:002018-04-21T13:27:49.339-07:00Redshift Experience<div dir="ltr" style="text-align: left;" trbidi="on">
Big Data - the keyword given to solutions that can handle massive amount of data usually in the petabyte or greater amount. There are several big data solutions out there and all of them have their unique characteristics which can be useful in different scenarios. I was looking into Cloudera's versions of Hadoop like Impala, Sentry and HBase. All these vary based on the use case. For some of my clients I have leveraged Amazon Redshift, Cassandra (and hopefully soon Apache Hadoop). The architecture of these systems differ but the end goal is the storage and processing of vast amounts of data down to second or milli second based result generation. Focusing on this aspect I am going to give a more detailed insight on Redshift which is a node based peta byte scaled database as well as a high level overview of what I recently implemented. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYdkQVv04FrFvEFFtvs6MqBGEAIMoH8B7tdVD0ET1JMf0IPYxPGkcKU-et-oQnbz2JkoiqLFgzop64wlAp3bcgfap7PHWIfSYjF4vkivFUf4y0eGyDpGuOMZui3NCNu8puyyANoQ/s1600/02-NodeRelationships.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYdkQVv04FrFvEFFtvs6MqBGEAIMoH8B7tdVD0ET1JMf0IPYxPGkcKU-et-oQnbz2JkoiqLFgzop64wlAp3bcgfap7PHWIfSYjF4vkivFUf4y0eGyDpGuOMZui3NCNu8puyyANoQ/s1600/02-NodeRelationships.png" width="320" /></a></div>
Note: The above diagram is from the Redshift Warehousing article (<a href="http://docs.aws.amazon.com/redshift/latest/dg/c_high_level_system_architecture.html)">http://docs.aws.amazon.com/redshift/latest/dg/c_high_level_system_architecture.html)</a><br />
<br />
If you pay close attention to the diagram above the compute nodes is responsible for all the data processing and query transactions on the database and the data nodes contain the actual node slices. The Leader Node is the service that monitors the data connections against the Redshift cluster and is also responsible for the query processing (almost like a syntax checker on the query and functions leveraged). It then transfers the query across to the Compute Nodes whose main responsibilty is to find the data slices from the data nodes and communicate with one another to determine the way the transaction needs to be executed. It is similar to a job broker except that this is more real time than non real time. <br />
<br />
It is similar to the analogy of using a bucket..... Consider this:<br />
<br />
You take a bucket and keep filling water into it, eventually the bucket get filled..... however what happens when there is an enormous amount of water that needs to be contained. Either grab a massive bucket or use multiple buckets to store the water (so the second option actually depicts the Redshift architecture....)<br />
<br />
The concept scaling up implies not only adding a bucket for storage but also a mechanism to ensure that the pipeline flow goes to an empty bucket which is nothing but out compute node. But there is the price of the bucket and the cost of the mechanism that is required to populate the bucket as well....<br />
<br />
The costing of data nodes and compute nodes are as follows:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2nT6g0ugfwc5hr7LS0UostURQtBt2QbsdB1FtvMo4Bnv7ZuVJW9PRNSb9yQgV1ZOTJQJwo_V8wojeUtY0_vX5ZSz9h1xrPgu2QKmZcvCYsnFDcbaKDQSx2RMi2zoXkca2LRC0FA/s1600/Capture.PNG" width="320" /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0se_VUkmlSlaVKa0rX0FCPW5LYVIW1rhcFQQxIGXc1POFotRJIW302nkAbRfvh_bonRbvOPxl40mYY1X5RegmO9hMOYE4NVKOwnBCFwUnkAsLZom73Eb0tU9Xi52iiEULDc9vqA/s1600/Capture1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0se_VUkmlSlaVKa0rX0FCPW5LYVIW1rhcFQQxIGXc1POFotRJIW302nkAbRfvh_bonRbvOPxl40mYY1X5RegmO9hMOYE4NVKOwnBCFwUnkAsLZom73Eb0tU9Xi52iiEULDc9vqA/s1600/Capture1.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
As the data node volume keeps increasing, reserve a data node and then utilize it as required. While creating the Redshift cluster ensure that the cluster type is Multi - Node and not Single-Node for large data related projects. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The tables are stored with a columnar partitioning algorithm on the different nodes governed by the distribution key of the respective table. The database used within Redshift is Postgres (the version keeps getting updated based on the release), this will be handled by Amazon. On a quick note whenever you are trying to add a node to the cluster, the database will go down so ensure that all your ETL's or import/export jobs are stopped during this time frame. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<strong>Even though daily snapshots are taken inside the Redshift cluster, a key note here is if one is ever going to delete the cluster and recreate it from a previous snapshot, ensure that a new cluster is created from the snapshot which is required and delete the first redshift cluster.</strong></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Even though Redshift has a lot of data integration tools partnerships, I would definitely recommend leveraging Talend ETL. I might give a breakdown of an Talend ETL in a upcoming article.</div>
<div class="separator" style="clear: both; text-align: left;">
For reporting on the other hand there are several analytics tools like Pentaho, Microstrategy, Tableau, Birst etc..., I would recommend using Tableau/Pentaho/Microstrategy for analytical dashboarding needs based on the cost consideration. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Let us get back to Redshift, the storage mechanism is similar to the Tabular vertipaq engine model that Microsoft leverages. And the processing is done using MPP i.e. Massive parallel Processing engines. The encryption is two fold, the first fold is at the database level where the Data Encryption Key encrypt data blocks in the cluster. Each data block is assigned a randomly-generated AES-256 key. These keys are encrypted by using the database key for the cluster. The other fold is at the data level or commonly known as encoding. I am just giving a flat table of what encoding needs to be used for what data type:-</div>
<table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse; width: 277px;">
<colgroup><col style="mso-width-alt: 7566; mso-width-source: userset; width: 160pt;" width="213"></col>
<col style="width: 48pt;" width="64"></col>
</colgroup><tbody>
<tr height="19" style="height: 14.4pt;">
<td class="xl65" height="19" style="background-color: transparent; border: 0.5pt solid windowtext; height: 14.4pt; width: 160pt;" width="213"><span style="font-family: "calibri";">int,date,datetime,
timestamps</span></td>
<td class="xl65" style="background-color: transparent; border-color: windowtext windowtext windowtext black; border-style: solid solid solid none; border-width: 0.5pt 0.5pt 0.5pt 0px; width: 48pt;" width="64"><span style="font-family: "calibri";">delta</span></td>
</tr>
<tr height="19" style="height: 14.4pt;">
<td class="xl65" height="19" style="background-color: transparent; border-color: black windowtext windowtext; border-style: none solid solid; border-width: 0px 0.5pt 0.5pt; height: 14.4pt;"><span style="font-family: "calibri";">varchar or
char upto 50 characters</span></td>
<td class="xl65" style="background-color: transparent; border-color: black windowtext windowtext black; border-style: none solid solid none; border-width: 0px 0.5pt 0.5pt 0px;"><span style="font-family: "calibri";">runlength</span></td>
</tr>
<tr height="19" style="height: 14.4pt;">
<td class="xl65" height="19" style="background-color: transparent; border-color: black windowtext windowtext; border-style: none solid solid; border-width: 0px 0.5pt 0.5pt; height: 14.4pt;"><span style="font-family: "calibri";">varchar<span style="mso-spacerun: yes;"> </span>> 50 characters</span></td>
<td class="xl65" style="background-color: transparent; border-color: black windowtext windowtext black; border-style: none solid solid none; border-width: 0px 0.5pt 0.5pt 0px;"><span style="font-family: "calibri";">text255</span></td>
</tr>
<tr height="19" style="height: 14.4pt;">
<td class="xl65" height="19" style="background-color: transparent; border-color: black windowtext windowtext; border-style: none solid solid; border-width: 0px 0.5pt 0.5pt; height: 14.4pt;"><span style="font-family: "calibri";">smallint,decimal</span></td>
<td class="xl65" style="background-color: transparent; border-color: black windowtext windowtext black; border-style: none solid solid none; border-width: 0px 0.5pt 0.5pt 0px;"><span style="font-family: "calibri";">Mostly8</span></td>
</tr>
<tr height="19" style="height: 14.4pt;">
<td class="xl65" height="19" style="background-color: transparent; border-color: black windowtext windowtext; border-style: none solid solid; border-width: 0px 0.5pt 0.5pt; height: 14.4pt;"><span style="font-family: "calibri";">Others</span></td>
<td class="xl65" style="background-color: transparent; border-color: black windowtext windowtext black; border-style: none solid solid none; border-width: 0px 0.5pt 0.5pt 0px;"><span style="font-family: "calibri";">Raw</span></td>
</tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The cons of using Redshift are as follows:</div>
<ul style="text-align: left;">
<li><div class="separator" style="clear: both; text-align: left;">
You need an Administrator to monitor the Redshift cluster at all times</div>
</li>
<li><div class="separator" style="clear: both; text-align: left;">
The power of Redshift lies in the manner the database is queried, so a SQL developer/DBA with understanding of the Redshift internals is definitely a must</div>
</li>
<li><div class="separator" style="clear: both; text-align: left;">
Upfront cost is slightly on the higher side but over a period in time the cost will be justified with more nodes being added to the cluster</div>
</li>
</ul>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Redshift allows both JDBC and ODBC drivers so definitely if you want to query the database. The Postgres driver can be found as follows:</div>
<div class="separator" style="clear: both; text-align: left;">
<a href="http://jdbc.postgresql.org/download.html">http://jdbc.postgresql.org/download.html</a></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="http://www.postgresql.org/ftp/odbc/versions/msi/">http://www.postgresql.org/ftp/odbc/versions/msi/</a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Like any other traditional RDBMS, if there is a constant amount of queries running on the same table, the locking of the table is inevitable, so care must be taken to ensure that the tables are not locked using the following statement:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<strong><span style="color: blue;"> -->select * from</span> <span style="color: lime;">STL_TR_CONFLICT</span></strong></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Also ensuring that the statistics are built on a regular basis....run the following command:</div>
<div class="separator" style="clear: both; text-align: left;">
<strong><span style="color: blue;"> -->Analyze</span> <span style="color: lime;">Table_Name</span></strong></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="color: black;">for all the tables on which queries are performed on a periodic basis.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Also enable auditing on the cluster using the Amazon Redshift console. </div>
<div class="separator" style="clear: both; text-align: left;">
A good practice is to ensure that the datawarehouse schema conforms to a Star schema to improve the query performance.</div>
<div class="separator" style="clear: both; text-align: left;">
This is it for now but stay tuned to more updates in this regard!!!</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-13671504854791963662014-05-08T14:17:00.000-07:002014-05-08T14:29:14.502-07:00SSAS Cube issues ..... Incorrect Measure Values<div dir="ltr" style="text-align: left;" trbidi="on">
Have you ever noticed that your measure value in SSAS do not correspond to the value in the data warehouse.... this is a big hindrance....because one is wasting precious development time in <br>extrapolating data between the cube and the warehouse. I am just going to create a checklist for issues to look at really quick if one does come across this issue!!!<br>
<br>
Problem Statement:
<br>
Let us consider a Fact table called FactInternetSales with a fact called internetsalesamount
<br>
<table border="1">
<tr>
<td bgcolor="white">
<font color="black">
select sum(internetsalesamount) from FactInternetSales;
</font>
</td>
</tr>
</table>
<br>
Let us say that this value is 25000.<br>
Now if we run this against the cube .... lets call it Sales.<br>
<table border="1">
<tr>
<td bgcolor="white">
<font color="black">
select measures].[internetsalesamount] on 0 from Sales;<br>
</font>
</td>
</tr>
</table>
Now the value returned from this query is 510......<br>
Why is this happening? Now just follow the following checklist to ensure that you can rectify this issue as soon as one possibly can....<br>
1. Go to the cube and in the [internetsalesamount] properties, change the aggregation value to count instead of sum. Validate whether the measure value matches the count in the datawarehouse i.e. select count(1) from FactInternetSales.... and ensure that the count matches.<br>
2. Check the dimensions that are slicing the cube and ensure that all the dimension attribute values have a 1:1 or 1: many mapping with the FactInternetSales. Majority of the time, there could be a dimension value missing in the dimension which is present in the cube. If this is the case, try and figure out why the dimension value is missing and add it back in.<br>
3. Sometimes the datatype used as part of the measure could also lead to such problems so just a quick cheat sheet to leverage:<br>
int (in datawarehouse) --> corresponding measure should be bigint<br>
decimal --> double <br>
4. Constant changes to the fact table..... now if the fact table constantly changes, it is best practice to change the processing model to ROLAP instead of the MOLAP (I know you end up with more processing time but nothing beats real time data especially for the analysts)<br>
5. Check the SQL query that is generated in populating the Fact in SSAS... this might be an inconclusive in majority of the scenarios but its still worth a shot<br>
6. Final step in the checklist, if all else fails try try and succeed........<br>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-73667438462888918032014-02-07T02:27:00.002-08:002014-02-18T11:17:09.545-08:00New CEO for Microsoft<div dir="ltr" style="text-align: left;" trbidi="on">
It was recently announced that Satya Nadella (a Microsoft veteran) has become the CEO of Microsoft. Even though I have a great sense of pride that a person of Indian origins or a first generation American immigrant of Indian origins is now the biggest and probably the most powerful person in the I.T. industry today, Microsoft placed a safe bet in naming Nadella as its CEO. I feel that a better choice would have been Sundar Pichai (VP @ Google) or even Robin Li ( The founder and CEO of Baidu). Of course from an experience point of view Nadella would need absolutely no introduction. His resume speaks for itself. He created and successfully handled a multitude of Microsoft's unique silo's or divisions. But infusing new ideas decoupled from the earlier thought process that Microsoft has always had would have paved for a new era when it comes to Microsoft revolutionizing the tech industry. This is a factor that will bear a brunt in Microsoft's cogs because clearly Nadella has been part of this thought process that has been detrimental to Microsoft and probably will not deviate from this (Old habits die hard).
Another big let down was the fact that my idol Bill Gates is back in a technical role. It just shows the level of confidence that the old guard at Microsoft has for the new CEO. Of course Microsoft has pushed it across as a highlight of Nadella becoming the CEO, since the legendary Bill Gates with all his experience of building an empire is back at the tech table. But to me this is more of a smoke screen than anything else. This made Nadella look a bit weak in retrospect with what should have considered to have been a tribute to the old era. Bill Gates and Steve Ballmer could have come out with their heads held high with a great sense of achievement of building a global entity from scratch and handing over the reigns to Nadella but this does not remain the case as Mr. Gates is back in a full time capacity and Ballmer would definitely be present for an interim period till Nadella settles down. This is the reason Microsoft should have gone with a fresh CEO, a person who can enable Microsoft's key strengths as well as infuse new ideas. I was rooting for Sundar since his name was dropped into the hat. Who knows this might have been the handshake that the two biggest giants in the industry Google and Microsoft might have required. Another person who I thought of when Microsoft was searching for a new CEO was Mark Zuckerburg. Sure he has a Facebook to handle, but just imagine if he was named the CEO of Microsoft and Facebook would no longer remain a unique identity but becomes a part of Microsoft. That truly would have been an earth shattering event. Two giants coming together as one integrated unit which would continue to create magic ( a dream that never became reality). These are my thoughts and I felt like expressing them but you never know maybe the Nadella-Gates combination might become a <b>"Six"</b> (as they call it in cricket).
<br /></div>Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-32206249694662176262014-01-21T16:31:00.003-08:002014-01-21T17:12:18.769-08:00SSIS SFTP handling using WinSCP<div dir="ltr" style="text-align: left;" trbidi="on">
Just a follow on to the earlier post.....
<span style="color: #008800; font-weight: bold"> <br/>
try {<br/>
string file = Dts.Variables["strLocalDirectory"].Value.ToString() + "\\" + Dts.Variables["strLocalFile"].Value.ToString();<br/>
string username = Dts.Variables["strFTPUserName"].Value.ToString();<br/>
string password = Dts.Variables["strFTPPassword"].Value.ToString();<br/>
// Setup session options<br/>
SessionOptions sessionOptions = new SessionOptions {<br/>
HostName = Dts.Variables["strFTPHost"].Value.ToString(),<br/>
UserName = username,<br/>
Password = password,<br/>
<span style="color: yellow; font-weight: bold">
Protocol = Protocol.Sftp,<br/></span>
PortNumber = int.Parse(Dts.Variables["strFTPPort"].Value.ToString()),<br/>
FtpMode = FtpMode.Active,<br/>
FtpSecure = FtpSecure.None,<br/>
<span style="color: yellow; font-weight: bold">
SshHostKeyFingerprint = Dts.Variables["strFTPFingerprint"].Value.ToString()<br/>
</span>
};
using (Session session = new Session()) {<br/>
session.ExecutablePath = Dts.Variables["strWinSCPExecutable"].Value.ToString();<br/>
// Connect<br/>
session.Open(sessionOptions);<br/>
// Upload files<br/>
TransferOptions transferOptions = new TransferOptions();<br/>
transferOptions.TransferMode = TransferMode.Binary;<br/>
TransferOperationResult transferResult;<br/>
transferResult = session.PutFiles(file, Dts.Variables["strRemotePath"].Value.ToString()); //./<br/>
}<br/>
}<br/>
catch (Exception e) {<br/>
Dts.Variables["strExceptions"].Value = e.Message + ":" + e.StackTrace;<br/>
Dts.TaskResult = (int)ScriptResults.Failure;<br/>
}<br/>
</span>
<br /></div>Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-71362166637863512352014-01-17T13:41:00.000-08:002014-01-17T13:49:22.322-08:00SSIS FTPS File handlingThe following scripttask showcases both the FTP and FTPS based connections for uploading a file (it can be modified to perform more options). My earlier approach was to leverage the FTPWebRequest but the "AUTH SSL" command was taking a huge amount of time so decided to fall back on using WinSCP instead. Also ensure that the WinSCP.exe is added to the path environment variable and the WinSCP.dll is placed in the GAC.
<span style="color: #008800; font-weight: bold"> <br/>
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase <br/>
{ <br/>
public void Main() <br/>
{ <br/>
/*
* Traditional FTP
*/ <br/>
/*ConnectionManager mgr = Dts.Connections.Add("FTP");
try {
mgr.Properties["ServerName"].SetValue(mgr, "ftp://"+Dts.Variables["strFTPHost"].Value.ToString());
mgr.Properties["ServerUserName"].SetValue(mgr, Dts.Variables["strFTPUserName"].Value.ToString());
mgr.Properties["ServerPassword"].SetValue(mgr, Dts.Variables["strFTPPassword"].Value.ToString());
mgr.Properties["ServerPort"].SetValue(mgr, Dts.Variables["strFTPPort"].Value.ToString());
mgr.Properties["Timeout"].SetValue(mgr, Dts.Variables["strFTPTimeout"].Value.ToString());
mgr.Properties["UsePassiveMode"].SetValue(mgr, true);
mgr.ConnectionString = Dts.Variables["strFTPHost"].Value.ToString() + ":" + Dts.Variables["strFTPPort"].Value.ToString();
//string conn = Dts.Connections["FTP Connection Manager"].ConnectionString;
FtpClientConnection ftpConn = new FtpClientConnection(mgr.AcquireConnection(null));
ftpConn.Connect();
string[] fileLocation = new string[1];
fileLocation[0] = Dts.Variables["strLocalDirectory"].Value.ToString()+"\\"+Dts.Variables["strLocalFile"].Value.ToString();
ftpConn.SendFiles(fileLocation, Dts.Variables["strRemotePath"].Value.ToString(), true, false);
ftpConn.Close();
}
catch (Exception exp) {
Dts.Variables["strExceptions"].Value = exp.Message + ":" + exp.StackTrace;
}*/ <br/>
/*
* FTPS using WinSCP
*/ <br/>
try {
string file = Dts.Variables["strLocalDirectory"].Value.ToString() + "\\" + Dts.Variables["strLocalFile"].Value.ToString(); <br/>
string username = Dts.Variables["strFTPUserName"].Value.ToString(); <br/>
string password = Dts.Variables["strFTPPassword"].Value.ToString(); <br/>
//string.Format("Running sync with {0}", username).Dump(DateTime.Now.ToString()); <br/>
// Setup session options
SessionOptions sessionOptions = new SessionOptions { <br/>
HostName = Dts.Variables["strFTPHost"].Value.ToString(), <br/>
UserName = username, <br/>
Password = password, <br/>
Protocol = Protocol.Ftp, <br/>
PortNumber = Dts.Variables["strFTPPort"].Value.ToString(), <br/>
FtpMode = FtpMode.Passive, <br/>
FtpSecure = FtpSecure.Implicit, <br/>
TlsHostCertificateFingerprint = Dts.Variables["strFTPFingerprint"].Value.ToString() <br/>
};
<br/>
using (Session session = new Session()) { <br/>
session.ExecutablePath = Dts.Variables["strWinSCPExecutable"].Value.ToString(); <br/>
// Connect
session.Open(sessionOptions); <br/>
// Upload files <br/>
TransferOptions transferOptions = new TransferOptions(); <br/>
transferOptions.TransferMode = TransferMode.Binary; <br/>
TransferOperationResult transferResult; <br/>
transferResult = session.PutFiles(file, Dts.Variables["strRemotePath"].Value.ToString()); <br/>
// Throw on any error <br/>
transferResult.Check(); <br/>
} <br/>
} <br/>
catch (Exception e) { <br/>
Dts.Variables["strExceptions"].Value = e.Message + ":" + e.StackTrace; <br/>
Dts.TaskResult = (int)ScriptResults.Failure; <br/>
}
Dts.TaskResult = (int)ScriptResults.Success; <br/>
}
</span>
<br/>
The variables used in the above code is self explanatory.... Remember to get the thumbprint using WinSCP initially to get the certificate or check with the FTP system admin.Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-87642745301581979942013-09-19T16:12:00.000-07:002013-09-19T16:19:47.644-07:00Rhino - ETL<div dir="ltr" style="text-align: left;" trbidi="on">
Using Rhino-ETL ( a C# based) framework for developing standard ETL's is pretty easy and one can do a lot of fun stuff with the underlying source data. I just wrote up a quick console app to generate data into a text file and push the same data into a table in SQL Server as well as an external file. Here are the following steps (for the external file push):<br />
1. Create a new C# console application solution in Visual Studio.<br />
2. Target the .Net framework as shown in the below screen shot in your project properties:-<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxXzPVkIhstxotPhA_0osAkPxZLJ5G6ulsbvT5vXl_95YkOnFmT5bTY-Tn2bq6BMdqLdAn5HfTV4mJ74cby5LlI8DDl2W709lGJzSB9DZp48afCeJRm153Xn7_loKHIQE7VB8CTw/s1600/p1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxXzPVkIhstxotPhA_0osAkPxZLJ5G6ulsbvT5vXl_95YkOnFmT5bTY-Tn2bq6BMdqLdAn5HfTV4mJ74cby5LlI8DDl2W709lGJzSB9DZp48afCeJRm153Xn7_loKHIQE7VB8CTw/s320/p1.png" width="320" /></a></div>
<br />
3. Create 3 Sub Folders underneath your project as shown in the following screen shot<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUjVg9abi_mSEBluQi9BXD2bPt___fEfphz3nf0YKdfgLA08FTYKgpgTCL6fB47d5jWFUdnbnshAusPyy2MTQyWXBzWnsZmRAfEiMcOqfoksiqT3d-i9jK4JhWZugdozdbpvpGHg/s1600/p2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUjVg9abi_mSEBluQi9BXD2bPt___fEfphz3nf0YKdfgLA08FTYKgpgTCL6fB47d5jWFUdnbnshAusPyy2MTQyWXBzWnsZmRAfEiMcOqfoksiqT3d-i9jK4JhWZugdozdbpvpGHg/s1600/p2.png" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<b><u>DataObjects</u></b> --> Contains the class files associated with each and every table/file in your environment. Example:- if your source file contains student data, then you would create a class file called Student with the individual properties (in relation to the properties) exposed (nouns).</div>
<div class="separator" style="clear: both; text-align: left;">
<b><u>Operations</u></b>--> This primarily contains the class files that contain the activities (adjectives) that need to be performed on the DataObjects. Example:- Writing the Student Data to a database, Reading the Student data from a file etc.</div>
<div class="separator" style="clear: both; text-align: left;">
<u style="font-weight: bold;">WorkFolder</u>--> Contains the external file sources to interact with. Example:- a flat file, a csv or a tsv. In this case it will be student.txt.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Lets write some code to insert a student record from a flat file into another flat file.......(as simple as it sounds)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
4. Create a class file called StudentRecord.cs (pipe delimited) and declare the required entity attributes as shown in the following code snippet:-</div>
<div class="separator" style="clear: both; text-align: left;">
<script src="https://gist.github.com/dynamicishwar/6630826.js"></script></div>
<div class="separator" style="clear: both; text-align: left;">
Contains records in the following manner (student.txt)</div>
<div class="separator" style="clear: both; text-align: left;">
StudentId|StudentName|StudentAddress|StudentClassId|StudentMarks //header</div>
<div class="separator" style="clear: both; text-align: left;">
1|Ishwar|TestAddress|1|85 //row</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
5. Create a class file called NewStudentRecord (which contains the attributes that need to be transferred to the new file)</div>
<div class="separator" style="clear: both; text-align: left;">
<script src="https://gist.github.com/dynamicishwar/6630938.js"></script></div>
<div class="separator" style="clear: both; text-align: left;">
This will be outputted in the following manner(tab separated)</div>
<div class="separator" style="clear: both; text-align: left;">
StudentId\tStudentName\tStudentAddress\tStudentClass</div>
<div class="separator" style="clear: both; text-align: left;">
1\tIshwar\tTestAddress\t1</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Let us now create the action called student write i.e. Let us go about writing this out to an output file called studentoutput.txt and I am creating a new C# class file called StudentWriteFile which will be as shown in the following code snippet:-</div>
<div class="separator" style="clear: both;">
<script src="https://gist.github.com/dynamicishwar/6630861.js"></script></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Now let us go about writing the main program... create the main program in the following manner:-</div>
<div class="separator" style="clear: both; text-align: left;">
<script src="https://gist.github.com/dynamicishwar/6631118.js"></script></div>
<div class="separator" style="clear: both; text-align: left;">
The setting's values basically point to the settings files that I have created which contains the absolute path of the student.txt and the studentoutput.txt files.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
After which in your main just initialize the MainProgram in the following manner:-</div>
<div class="separator" style="clear: both; text-align: left;">
new MainProgram().Execute();</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
and you will have your first rhino-etl to rock and roll with......</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br /></div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com2tag:blogger.com,1999:blog-32547094.post-3987020480628426152013-07-02T19:05:00.002-07:002013-07-02T19:05:41.297-07:00Javascripting libraries and frameworks.....Which is the best??<div dir="ltr" style="text-align: left;" trbidi="on">
Recently got back into active web application development after quite a long time (4-5 years)..... It is always fun working on the UI layer except for the fact that everyone has their own opinions when it comes to look and feel. Anyways this article is to stress some of the javascript libraries that I am currently evaluating and trying to leverage for a new website....<br />
<br />
1. Backbone.js--> I just loved using this library, easy construction of Models - Backbone.Model.extend({ Model definition }) and views - Backbone.View.extend({view declaration}). It seems to not adhere to any specific pattern. A lot of sites claim this to be a traditional MVC structure but I disagree because of the fact that this library can cater to other popular patterns out there in the market today. I have decided to settle down on this for my web site just because of the code base and the ease of use of writing backbone. It supports RESTful based routing as well.... so overall just one word for it awesome... Also on a quick note it seems to have a great tie in with another library that I will be leveraging called D3.js (Data driven documents) which is a phenomenal repository of different data representation factors.<br />
<br />
2. Knockout --> Another library but supports the ever popular Microsoft's MVVM pattern. Knockout is another cool library to leverage and works extremely well with SPA based apps. Knockout binds the elements with JSON payloads phenomenally well and is something that Microsoft has been leveraging in most of their SPA based apps. Cool tutorials also in place @ <a href="http://learn.knockoutjs.com/#/?tutorial=intro">http://learn.knockoutjs.com/#/?tutorial=intro</a><br />
<br />
3. Dojo --> Have not played around with this framework yet but a lot of people currently do leverage (not as many as backbone maybe a similar to knockout's user base) Dojo. Really neat name.... Concept of modular based programming in place. Everything is neatly structured with both the properties and methods exposed by the API code base.....Also has its own charting tools and a widget library (especially for card building in web sites) available.<br />
<br />
The other popular javascript libraries out there are as follows:<br />
Spine (another version of knockout) and Canjs<br />
Popular frameworks out there are Angular js, Meteor and Batman.<br />
<br />
A great site for referencing the various libraries and frameworks available out there is Steven Anderson's blog site @ <a href="http://blog.stevensanderson.com/2012/08/01/rich-javascript-applications-the-seven-frameworks-throne-of-js-2012/">http://blog.stevensanderson.com/2012/08/01/rich-javascript-applications-the-seven-frameworks-throne-of-js-2012/</a><br />
<a href="http://blog.stevensanderson.com/2012/08/01/rich-javascript-applications-the-seven-frameworks-throne-of-js-2012/"></a><br />
<br />
Just to add to the list of charting libraries out there:<br />
1. Rgraph<br />
2. Flot2<br />
3. Google graphs<br />
4. D3.js<br />
5. Dojo<br />
6. High Charts<br />
<br />
Check out the following site for more graphing tools out there in the market:<br />
<a href="http://mannaio.altervista.org/design/interactive-javascript-charts/?doing_wp_cron=1372816883.0944008827209472656250">http://mannaio.altervista.org/design/interactive-javascript-charts/?doing_wp_cron=1372816883.0944008827209472656250</a></div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com0tag:blogger.com,1999:blog-32547094.post-58295266677103198122013-05-14T19:27:00.001-07:002013-05-14T19:30:10.276-07:00An Agile process evaluation<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
In
the recent past, I have been working as a scrum master for one of the projects.
And since there were two folds to this project, a portion being handled onsite
as well as a portion being handled offshore, there were initial hiccups that
gradually deteriorated away. But here comes the crux of the problem and where
an Agile process has to be evaluated.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
We had weekly sprints where a subset of
tasks were to be completed and we have a daily scrum between the offshore and
onsite team. But the onsite team were persistent on a daily mail sent with the
changes being addressed. Now the point of the scrum is to take into account who,
what and where the changes have to be made based on the product backlog. Now
having status mails as part of the Agile process is indeed a flaw in the
fundamental flow. After pointing out this fact, I was counter attacked with
several points (some of which were valid to have a status update email) and
finally left to comply. Now here comes the secondary flaw and something which
had a major impact in sprint. The onsite team demanded daily code
reviews of the work done offshore. Now this caused a major butterfly effect in
the scrum process. First the offshore team needs to comply to the tasks that
need to be performed and then perform a code merge of all the developer's changes
(again done by one of the team members) which put a lot of stress on the team
member doing the merge. I tried to stream line this by asking different members
to perform this on a daily basis, but eventually it could not be done in this
fashion as well as even suggested a bi-weekly drop which was given a thumbs
down. This led to a lot of down time and a morale let down as even I could not
prevent this from happening as the scrum master of the team. I could not shield
the team from this negative impact. And the other intake I could make was that
the code sent for review was not being reviewed on a daily basis (this is the
icing on the cake). I on the other hand had to ensure that a few deliverables
on my plate did meet the deadline and could not help the offshore team in
making changes to the Agile workflow we had in place. Then came the impact of
managing the project with TFS (which I definitely had my eyes on) since it
became easier to track down the deficits as well as generate charts to figure out
where we were heading towards each sprint. But this was turned down by let's
say the new product owner who wanted to do this via project plans. With all
this and all my say not being looked at, we struggled to get the project to
production ready with very minimal glitches.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
I think the biggest
mistake was my accepting responsibility as the scrum master. The scrum master
must be the most strong willed member of the team and should say "NO"
when the moment calls for it. But then comes to adapting to the organization's
work culture which ultimately led to the undoing of the Agile process. I gave
in more to the culture of the organization than the needs of the project which
might happen in the future as well. But the next time I will more geared and in
case I see the red lights propping up, I am going to step down as the scrum
master if I were made one. Also in the case of the pigs and chickens analogy,
the chickens had the major say in this case which again is an Agile flaw. But
this has left me with quite a few questions unanswered:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11pt; margin-bottom: 0in; margin-left: 0.375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-size: 11pt;">Corporate Culture VS Agile
project management. Which needs to be given more priority and Why?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-size: 11pt;">Do all the team members have
equal status in the Agile process or are there scenario's where certain
team members should have more pull?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-size: 11pt;">If point 2 holds, what could
be scenarios this could occur?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-size: 11pt;">Product owner VS Scrum Master
in making product design changes based on the sprint timing. Who should
have more hold?</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
Based
on my understanding of how Agile works and some of the books I read, I get
varying answers to these questions. I think it something that I need to ponder
on and figure out an approach myself. Stay tuned to a future article about my
insights into such situations….</div>
</div>
Ishwarhttp://www.blogger.com/profile/15762644747356541809noreply@blogger.com1