Quantcast
Channel: Jose Barreto's Blog
Viewing all 47 articles
Browse latest View live

Using PowerShell to generate a large test CSV file with random data

$
0
0

I recently posted a new blog that shows how to load a very large CSV file into Excel, breaking the limit of 1 million rows in a single Excel sheet. If you haven’t seen it, you should check it out: Loading CSV/text files with more than a million rows into Excel.

One of the challenges for me while writing that blog post was finding sample data to demonstrate that Excel feature. I have a few files that are that big from work, but obviously I could not share those publicly. I also tried to find some sample data on public sites like http://data.gov but I could not find anything with more than 1 million rows.

The only option I had left was creating a sample file myself with some random data. I created a simple PowerShell script to create a file with a few columns filled with random data. You would think that it would be straightforward to do that, but my first version, which wrote one line to the file at a time, would take hours to generate a file with 2 million rows. Not good enough.

I optimized it by creating a thousand rows in memory and writing that batch to the file in one operation. Repeating that 2,000 times gave me a file with two million rows in under 20 minutes. It included 4 columns and about 36 characters per row. I ended up with a file size of 68.8 MB (72,144,471 bytes).

Note that I had to specify the encoding, since the default encoding in a regular PowerShell output would create double-byte characters and would make the resulting file twice as large.

Below is the PowerShell script, both as text and a picture with the color coding from the PowerShell ISE.

$Start = Get-Date
$BaseDate = (Get-Date "2016/12/31 12:59:59")
$FullYear = 366*24*60*60

$File = ”c:\users\jose\file.csv”
"Date,Customer,Type,Value" | Out-File -FilePath $File -Encoding utf8

1..2000 | % {

  Write-Progress -Activity “Generating File” -PercentComplete ($_/20)
  $Lines = ""
  1..1000 | % {
    $Dt = $BaseDate.AddSeconds(-(Get-Random $FullYear))
    $Ct = (Get-Random 100)
    if ((Get-Random 5) -lt 4) {$Ty="Sale"} else { $Ty="Return"}
    $Vl = (Get-Random 100000) / 100
    $Lines += [string]$Dt + "," + [string]$Ct + "," + $Ty + "," + [string]$Vl + [char]10 + [char]13
  }

  $Lines | Out-File -FilePath $File -Encoding utf8 -Append
}

$End = Get-Date
"Started at $Start and ended at $End"
$Diff = ($End-$Start).TotalSeconds
"Processing took $Diff seconds"

large-file-powershell


My top tweets from 2017

$
0
0

These are my top tweets from each month in 2017, according to https://analytics.twitter.com.

January 2017 

Top Tweet (earned 4,849 impressions)

Free Azure training via Microsoft on Open edX. These are massive open online courses (MOOCs). openedx.microsoft.com pic.twitter.com/hdzJyrlEG3

Top media Tweet (earned 2,831 impressions)

Ars Technica UK: Intel Core i7-7700K Kaby Lake review: Is the desktop CPU dead? arstechnica.co.uk/gadgets/2017/0… pic.twitter.com/xccsAxkH8R

February 2017

Top Tweet (earned 5,730 impressions)

Glassdoor: Data scientist, DevOps engineer, and data engineer are the best three jobs in America for 2017. techrepublic.com/article/report…

 Top media Tweet (earned 1,951 impressions)

There is a simple way to load a CSV file with over 1 million rows into Excel and create pivot tables from it. Writing a blog about it... pic.twitter.com/t0wstNlm8l

March 2017

Top Tweet (earned 2,654 impressions)

New blog by @ClausJor : To RDMA, or not to RDMA – that is the question blogs.technet.microsoft.com/filecab/2017/0… pic.twitter.com/aXp70BKIQw

Top media Tweet (earned 2,654 impressions)

Error Code 0x80070003 during Windows 10 Update? It might help to clean up \Windows\SoftwareDistribution\Download answers.microsoft.com/en-us/windows/… pic.twitter.com/LQkirdAy8S

April 2017

Top Tweet (earned 1,939 impressions)

Just received the agenda for our team meeting this morning: pic.twitter.com/glHWhRtuPW

Top media Tweet (earned 1,347 impressions)

Book: CEO Satya Nadella tells Microsoft's story, tracing his journey from childhood to leading digital era changes. news.microsoft.com/hitrefresh/ pic.twitter.com/UBHwY8sGsR

May 2017

Top Tweet (earned 2,128 impressions)

Build content will be streaming next week via Channel 9 channel9.msdn.com/Events/Build/2… pic.twitter.com/lFJUv0e8ak

Top media Tweet (earned 1,415 impressions)

Microsoft Security Response Center: Customer guidance for WannaCrypt attacks. blogs.technet.microsoft.com/msrc/2017/05/1… pic.twitter.com/y0NYkRepur

June 2017

Top Tweet (earned 1,497 impressions)

Do you have an idea on how to improve OneDrive? Share you feedback at onedrive.uservoice.com You can also vote on ideas already posted... pic.twitter.com/g2RICvbVHC

Top media Tweet
(earned 854 impressions)

High Efficiency Image File Format (HEIF) is an image file format by the MPEG (MPEG-H Part 12, ISO/IEC 23008-12) en.wikipedia.org/wiki/High_Effi… pic.twitter.com/XhnNMnQ5Ie

July 2017

Top Tweet (earned 1,572 impressions)

I typed on a ZX81 and a TRS80 yesterday... pic.twitter.com/6zOr7TWJlT

Top media Tweet (earned 1,179 impressions)

On-Demand Sessions from Microsoft Inspire coming soon! partner.microsoft.com/en-us/inspire/… pic.twitter.com/m9dCJj6okE

August 2017

Top Tweet (earned 2,248 impressions)

Azure Reference Architecture: Run a high availability SharePoint Server 2016 farm in Azure docs.microsoft.com/en-us/azure/ar…pic.twitter.com/jf2gdOVtLn

Top media Tweet (earned 1,725 impressions)

Understanding SSD endurance: DWPD, TBW, and the minimum recommended for Storage Spaces Direct by @CosmosDarwin blogs.technet.microsoft.com/filecab/2017/0… pic.twitter.com/OxNzwdUHo5

September 2017

Top Tweet (earned 5,125 impressions)

Ignite on-demand session videos starting to become available. Check them out at myignite.microsoft.com/videos pic.twitter.com/0A3Iqwauj1

Top media Tweet (earned 2,349 impressions)

Windows Server Blog: Now available: Windows Server 2016 Security Guide! blogs.technet.microsoft.com/windowsserver/… pic.twitter.com/eGyMuPF8Cm

October 2017

Top Tweet (earned 1,132 impressions)

Celebrating my 15th anniversary at Microsoft. Time flies when you're having fun... pic.twitter.com/kzJV1Fnon1

Top media Tweet (earned 992 impressions)

Windows 10's "Controlled Folder Access" Anti-Ransomware Feature Is Now Live bleepingcomputer.com/news/microsoft… pic.twitter.com/MkYPlT3Qro

November 2017

Top Tweet (earned 3,808 impressions)

Got my 15-year anniversary crystal today... pic.twitter.com/RSRFIhpTYE

Top media Tweet (earned 899 impressions)

A Quick Guide to OneDrive in Windows 10 via #MakeUseOf makeuseof.com/tag/quick-guid… pic.twitter.com/AF7d2dIl59

December 2017

Top Tweet (earned 1,252 impressions)

Azure Blog: Cloud storage now more affordable: Announcing general availability of Azure Archive Storage by Kumail Hussain azure.microsoft.com/en-us/blog/clo… pic.twitter.com/MY1rOz0bq0

Top media Tweet (earned 1,022 impressions)

You can sign into your OneDrive personal account using only the Microsoft Authenticator app on your phone, no password required. docs.microsoft.com/en-us/azure/mu… Tip: Get started by adding the Microsoft Account from the mobile app... pic.twitter.com/8icjQYmn4P

Path to the Server (An Ode to SMB Multichannel)

$
0
0

Read while listening to https://www.youtube.com/watch?v=iXQUu5Dti4g 

 

Path to the Server
On Ode to SMB Multichannel
by Jose Barreto

 

There’s a client somewhere looking for file shares
And it’s finding a path to the server
When it gets there it knows it can read, write and close
With a word it can get what it came for

Ooh, ooh, and it’s finding a path to the server

DNS is involved, but it needs to resolve
‘Cause you know sometimes names have two IPs.
In a dim server room, there is an admin who says:
Sometimes not all our addresses are given.

Ooh, it makes me wonder,
Ooh, it makes me wonder.

There’s a feeling, a twitch, when I look at the switch,
And my clients are crying for bandwidth.
In my thoughts I have seen
packets route out and in,
And the voices of those who stand waiting.

Ooh, it makes me wonder,
Ooh, really makes me wonder.

And it’s whispered to me if on SMB 3
Then the prot’col will lead us to reason.
And a new day will dawn for those file servers
And the clients will echo with laughter.

If you see a few more connections,
Don’t be alarmed now,
It’s just some multichannel action.
Yes, there are two paths you can go by but if one’s gone
There’s still time to change the queue you’re on.

And it makes me wonder.
Ooooooh…

You server is humming, speed is twofold,
In case you don’t know:
You are now balancing your net load.

Dear client, you’re now multipathing,
And just one more thing:
You will recover when your network fails.

Now we can read and write much more,
Our bandwidth higher than before,
There goes the server we all know.
Who perform betters and wants to show
How everything’s still just a share
And if you look it really hard
You’ll see pieces that we shard.
When there are multiple of all you got,
You’ll see a fail but you won’t stop.

And it’s finding the path to… the server…

Links about OneDrive Ransomware Detection and OneDrive Files Restore

$
0
0

Announcements

Office Blogs: Defend yourself from cybercrime with new Office 365 capabilities
By Kirk Koenigsbauer, Corporate Vice President for Office, Microsoft
https://blogs.office.com/en-us/2018/04/05/defend-yourself-from-cybercrime-with-new-office-365-capabilities/

OneDrive UserVoice: Provide Ransomware detection and recovery
By Douglas Pearce, Group Program Manager, Microsoft
https://onedrive.uservoice.com/forums/262982-onedrive/suggestions/12289380-provide-ransomware-detection-and-recovery

New support articles

Ransomware detection and recovering your files
https://support.office.com/en-us/article/ransomware-detection-and-recovering-your-files-0d90ec50-6bfd-40f4-acc7-b8c12c73637f

Restore your OneDrive files (all restore options)
https://support.office.com/en-us/article/restore-your-onedrive-files-48cc6a72-1aca-412c-a670-e6f5b563c1c6

Restore your OneDrive (Files Restore)
https://support.office.com/en-us/article/restore-your-onedrive-fa231298-759d-41cf-bcd0-25ac53eb8a15

News sites

The Verge: Microsoft adds ransomware protection and file restore to OneDrive cloud storage
Outlook.com also gets encrypted email support
By Tom Warren
https://www.theverge.com/2018/4/5/17201660/microsoft-onedrive-files-restore-feature-ransomware-protection

ZDNet: Microsoft to add ransomware protection tools, advanced Outlook.com encryption for consumers
Microsoft is bringing more of the OneDrive and Outlook security protection tools it offers its Office 365 business customers to consumers.
By Mary Jo Foley
https://www.zdnet.com/article/microsoft-to-add-ransomware-protection-tools-advanced-outlook-com-encryption-for-consumers/

Thurrott: OneDrive, Outlook, Other Office Products Get New Security Features
By Mehedi Hassan
https://www.thurrott.com/cloud/office-365/155848/onedrive-outlook-other-office-products-get-new-security-features

Tech Meme: Microsoft brings Files Restore to all OneDrive customers, letting users restore their entire OneDrive to a point in last 30 days, and adds ransomware detection  —  Microsoft is bringing more of the OneDrive and Outlook security protection tools it offers its Office 365 business customers to consumers.
https://www.techmeme.com/180405/p7#a180405p7

Engadget: Microsoft adds file protection and email encryption to Office 365
They're part of a new set of tools to guard users against cybercrime.
By David Lumb
https://www.engadget.com/2018/04/05/microsoft-office-365-add-cybersecurity-tools/

Windows Central: OneDrive Files Restore, ransomware detection, and more rolling out to Office 365
Several new security features are coming to OneDrive, Outlook, and more for Office 365 users.
By Dan Thorp-Lancaster
https://www.windowscentral.com/onedrive-files-restore-ransomware-detection-rolling-out-office-365

MS Power User: Microsoft brings OneDrive File Restore, ransomware detection and many new features to Office 365 users
By Pradeep Viswanathan

https://mspoweruser.com/microsoft-brings-onedrive-file-restore-ransomware-detection-and-many-new-features-to-office-365-users/

Digital Trends: Microsoft’s OneDrive now has your back in a ransomware attack
By Mark Coppock

https://www.digitaltrends.com/computing/microsoft-rolls-out-onedrive-files-restore-to-all-office-365-users/

On MSFT : OneDrive is getting new File Restore feature, ransomware protections with Office 365
By Laurent Giret
https://www.onmsft.com/news/onedrive-is-getting-new-file-restore-feature-ransomware-protections-with-office-365

Open sourcing SuperDir, a file manager for CP/M written in the 1980s

$
0
0

Microsoft has open sourced the 1990s File Manager for Windows 3.0 this week. You can check it out at https://github.com/Microsoft/winfile/

So I went even further and open sourced my 1980s File Manager for the CP/M OS written in Turbo Pascal, which a called SuperDir: https://github.com/Jose-Barreto/SuperDir

Back then, the OS had only 6 resident commands: DIR, ERA, REN, TYPE, SAVE and USER. Everything else was done through utilities. You can read more about it at this blog post: The good old days of CP/M 2.2.

The SuperDir file manager has a total of 10 functions:

  • Select drive
  • Show file
  • Print file
  • Rename file
  • Delete file
  • Undelete file
  • Copy file
  • Mark file read-only/read-write
  • Delete selected files
  • Copy selected files.

Note that reading the directory was done by actually reading the 128-byte disk sectors at the beginning of the disk and figuring out the 32-byte data structure describing each file. Names and extensions were fixed at 8.3 characters at the time. Also, there were no graphics in this system, so this was all done with standard ASCII characters on a 24x80 screen.

I'm not particularly proud of the pre-allocated arrays for storing the data. My sort function was also pretty lame. But hey, I wrote this in the 1980 when I was only starting to learn programming...

I took the time to add some comments in English, since the original had variables, function names and messages all in Portuguese. Now all I need is Turbo Pascal 2.0 compiler...

OneDrive Files Restore – User Guide

$
0
0

This guide is intended to help you use the new OneDrive Files Restore feature.

1. What is Files Restore?

Files Restore allows OneDrive users to restore their files and folders to a previous point in time. It provides a simple way to restore your entire OneDrive after a catastrophic event like a virus/ransomware attack or a user/application error that causes lots of changes to their OneDrive.

Files Restore

Restore builds on existing OneDrive capabilities like the Recycle Bin (allows files to be undeleted) and Version History (allows an individual file to be restored to a previous version). Restore integrates with Ransomware Detection, a new OneDrive feature that sends notification when signs of ransomware are found in your files.

NOTE: This guide is focused on OneDrive for consumers (also known as Personal OneDrive). OneDrive for Business (ODB) does offer Files Restore, but ODB is not the focus of this guide.

2. Files Restore Requirements

To use OneDrive Files Restore, you must:

3. How to use Files Restore?

To use OneDrive Files Restore, follow these steps:

Step 1 - Go to the Restore page (2 options)

  • Navigate to it from the main page
    • Start at the main OneDrive page at https://onedrive.com
    • Click on the gear icon in the upper right, then select “Options”
    • On the Options menu on the left, select “Restore my OneDrive”

Step 2 - Select a point in time (3 options)

  • Use the drop-down to select one of the presets (yesterday, one week ago, three weeks ago)
  • Use the drop-down to select a point of interest, if available (for example, a specific date and time when ransomware was detected)
  • Select “Custom date and time” in the drop down, then select a specific activity in the list.

Step 3 - Restore

  • Start the Restore – Click on the restore button

  • Monitor the restore – Wait on the restore page until it’s completed

Step 4 - Verify

  • Check if the files were properly restored to their previous condition.
  • You may also want to go back to the restore page to check on activities performed by the restore.

4. Known Issues

Here are the known issues for OneDrive Files Restore:

  • When navigating a very long list of activities, the full list might take a while to show. The activities do show up after some time.
  • When using the slider below the chart to select a specific day, the activities list might take a while to show that specific day. The activities do show up after some time.
  • For folder operations, the activities list shows a file icon for instead of a folder icon.
  • In small screens, the bars in the activity histogram might overlap.
  • After you start the restore, the progress indicator might jump from "0% complete" directly to "100% complete",  if you don't have too many activities to restore.

5. Frequently Asked Questions

1) Is this feature available for OneDrive for Business?

Yes, we have the Files Restore feature for OneDrive for Business, but this guide is intended to cover only the OneDrive version for consumers. The ODB version is already in production and you can learn more at https://techcommunity.microsoft.com/t5/OneDrive-Blog/Announcing-New-OneDrive-for-Business-feature-Files-Restore/ba-p/147436

2) How far back can I restore?

You can restore to any point in time in the last 30 days.

3) Can I undo a restore?

Yes. Just restore to the point in time before you performed the restore. Files Restore will keep a list of recent restores in the drop-down list, so you can easily go back to that point.

4) Can I restore after I empty the recycle bin?

Files Restore uses the recycle bin to go back in time. If the recycle bin was emptied or if any files were permanently deleted from the recycle bin, then those files cannot be restored.

5) If I sign-up for a free Office 365 trial today, can I immediately do a restore?

Yes.

6) Is there a direct URL for the restore page?

Yes. You can go directly to http://onedrive.live.com/?v=restore

7) How long does a restore take?

It depends on the number of activities that need to be restored. For a few thousand activities, a restore should take less than 10 minutes. For hundreds of thousands of activities, it will take a few hours.

8) During the restore, can I quit the browser and come back later?

Yes. The restore will continue and you will see a message the next time you access the OneDrive web site.

9) Can I run a restore from the OneDrive sync client or from the OneDrive mobile app?

The restore operation is always performed from the OneDrive web site. The activities restored will show up in all OneDrive clients, including Sync and mobile.

10) Can I run a restore from a mobile browser?

Yes. The Files Restore interface is mobile-friendly.

11) Can I change my files during a restore?

No. You should not make any changes to OneDrive until the restore finishes.

12) Can I restore just a folder? Can I restore just specific files?

Restore is designed to always bring your entire OneDrive back to a point in time. You can use the Recycle Bin or Version History to recover only specific files.

13) Can I have more than one Restore going at once?

No. Each OneDrive user can only perform one restore at a time. If you go to the Restore page and there’s already a Restore in progress, you will see the status of the current Restore.

14) Can I cancel or abort a restore?

Once a restore starts, you must wait until it finishes. You can see the status of the current Restore by going to the Restore page. Once the restore is completed, you can use a second restore to undo a previous one.

15) Will Restore revert actions like sharing and permissions?

No. Restore will only revert basic file and folder activities like create, delete, rename, update, move and copy.

6. More information

You can find the official content about this feature in the Restore your OneDrive page.

You can also see a demo of Files Restore on YouTube.

Please feel free to ask questions in the comments section below.

PowerShell script to check my backup by comparing file hashes

$
0
0

This sample PowerShell script compares the files in two different paths by calculating and comparing hashes for each file. I use it to compare my backup to the source data, after the backup is complete.

 

$SourcePath = "D:\"
$BackupPath = "F:\Backup 2018-05-27\"

Write-Progress -Activity "Getting File List" -PercentComplete 0
$FileList = DIR $SourcePath -Recurse -File

$Total = $FileList.Count
$Count = 0
$BadCount = 0
$FileList | % {

  $File = $_.FullName
  $Backup = $File.Replace($SourcePath, $BackupPath)

  Try { 
    $Match = (Get-FileHash $File).Hash -eq (Get-FileHash $Backup).Hash 
  } 
  Catch { 
    $Match = $false 
  }

  If (-not $Match) {
    $BadCount++
    "Hash mismatch: $File, $Backup"
  }

  $Count++
  If ($Count % 1000 -eq 0) {
    Write-Progress -Activity "File $Count of $Total" -PercentComplete ($Count/$Total*100) 
  }
}
Write-Progress -Activity "Checking Files" -Completed
"There were $BadCount bad files out of the $Count files checked"
Viewing all 47 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>