我们不能使用USQL Custom Code和usql上传文档/图像吗？

情况：我们在Azure数据湖分析中创建了数据库“CLSTrackOMeter”和表格“Customer_Information”。我们不能使用USQL Custom Code和usql上传文档/图像吗？

Customer_Information将图像的路径存储在staging文件夹中（现在我已经在类库中对源图像路径进行了硬编码）。

议程：使用来自CustInfo该值数据上传到Azure的数据湖店“Customer_Image”文件夹

试图解决 - 创建usql类库，使用.NET SDK的文件（可上传到在控制台应用程序中执行此类库），并部署在Azure数据库中。 - 增加了新USQL脚本，并引用该类库

称为类库usql脚本的CS文件

类的代码库

using Microsoft.Analytics.Interfaces; 
    using Microsoft.Analytics.Types.Sql; 
    using System; 
    using System.Collections.Generic; 
    using System.IO; 
    using System.Linq; 
    using System.Text; 
    using Microsoft.Azure.Management.DataLake.Store; 
    using Microsoft.Azure.Management.DataLake.Store.Models; 
    using Microsoft.IdentityModel.Clients.ActiveDirectory; 
    using Microsoft.Rest.Azure.Authentication; 
    using Microsoft.Azure.Management.DataLake.StoreUploader; 
    using System.Threading; 
    using System.Diagnostics; 
    using System.Collections; 
    using System.Threading.Tasks; 

    namespace USQLCSharpProject1 
    { 
     public static class Program 
     { 
      private static DataLakeStoreAccountManagementClient _adlsClient; 
      private static DataLakeStoreFileSystemManagementClient _adlsFileSystemClient; 

      private static string _adlsAccountName; 
      private static string _resourceGroupName; 
      private static string _location; 
      private static string _subId; 

      //private static void Main(string[] args) 
      public static string UploadFileWithS2S_WithClientSecret(string s) 
      { 

       try 
       { 
        _adlsAccountName = "<DATA-LAKE-STORE-NAME>"; // TODO: Replace this value with the name of your existing Data Lake Store account. 
        _resourceGroupName = "<RESOURCE-GROUP-NAME>"; // TODO: Replace this value with the name of the resource group containing your Data Lake Store account. 
        _location = "East US 2"; 
        _subId = "<SUBSCRIPTION-ID>"; 



        string localFolderPath = @"D:\Harry\PSR\study\TEST"; // TODO: Make sure this exists and can be overwritten. 
        string localFilePath = Path.Combine(localFolderPath, "fileTwo.txt"); // TODO: Make sure this exists and can be overwritten. 
        string remoteFolderPath = "/Samples/OUTPUT"; 
        //string remoteFilePath = Path.Combine(remoteFolderPath, "file.txt"); 


        // Service principal/appplication authentication with client secret/key 
// Use the client ID of an existing AAD "Web App" application. 
SynchronizationContext.SetSynchronizationContext(new SynchronizationContext()); 

var domain = "<AAD-directory-domain>"; 
var webApp_clientId = "<AAD-application-clientid>"; 
var clientSecret = "<AAD-application-client-secret>"; 
var clientCredential = new ClientCredential(webApp_clientId, clientSecret); 
var creds = ApplicationTokenProvider.LoginSilentAsync(domain,clientCredential).Result; 









        // Create client objects and set the subscription ID 
        _adlsClient = new DataLakeStoreAccountManagementClient(creds) { SubscriptionId = _subId }; 
        _adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds); 

        var parameters = new UploadParameters(localFolderPath, remoteFolderPath, _adlsAccountName, isOverwrite: true); // the default maxSegmentLength is 256M, we can set by ourself. 
        var frontend = new DataLakeStoreFrontEndAdapter(_adlsAccountName, _adlsFileSystemClient); 
        var uploader = new DataLakeStoreUploader(parameters, frontend); 
        uploader.Execute(); 
        return s; 
       } 

       catch (Exception ex) 
       { 
        return ""; 
       } 
      } 


     } 
    }

准则Usql

USE CLSTrackOMeter; 
REFERENCE ASSEMBLY USQLCSharpProject1; 
@result = 
    SELECT USQLUploadFile.myFirstClass.myFirstFunction(AgeGender)AS myFirstFunction_CB 
    FROM CLSTrackOMeter.dbo.Customer_Information;

个

USQL的码CS文件

using Microsoft.Analytics.Interfaces; 
using Microsoft.Analytics.Types.Sql; 
using System; 
using System.Collections.Generic; 
using System.IO; 
using System.Linq; 
using System.Text; 

namespace USQLUploadFile 
{ 
    public class myFirstClass 
    { 
     public static string myFirstFunction(string s) 
     { 
      try 
      { 

       string aa = USQLCSharpProject1.Program.UploadFileWithS2S_WithClientSecret("rajni"); 
       return aa; 

      } 
      catch (Exception ex) 
      { 
       return ""; 
      } 




     } 
    } 
}

项目图片

错误图片

错误在使用处理后的表情

USQL规范处理后的表情

USE CLSTrackOMeter; 
REFERENCE ASSEMBLY USQLCSharpProject1; 
    @result = SELECT AgeGender 
    FROM CLSTrackOMeter.dbo.Customer_Information; 

    @rs= 
    PROCESS @result 
    PRODUCE AgeGender 
    USING new USQLUploadFile.myFirstClass(); 


    OUTPUT @rs 
    TO "/output/Harry.csv" 
     USING Outputters.Csv();

USQL CS文件处理表达式代码

using Microsoft.Analytics.Interfaces; 
using Microsoft.Analytics.Types.Sql; 
using System; 
using System.Collections.Generic; 
using System.IO; 
using System.Linq; 
using System.Text; 

namespace USQLUploadFile 
{ 
    [SqlUserDefinedProcessor] 
    public class myFirstClass : IProcessor 
    { 
     public override IRow Process(IRow input, IUpdatableRow output) 
     { 
      try 
      { 
       string AgeGender = input.Get<string>("AgeGender"); 
       //USQLCSharpProject1.Class1 obj = new ClassLibrary1.Class1(); 
       //string aa = USQLCSharpProject1.Program.UploadFileWithS2S_WithClientSecret("rajni"); 
       //return aa; 
       string aa=USQLCSharpProject1.Program.UploadFileWithS2S_WithClientSecret("AgeGender"); 
       output.Set<string>("AgeGender", AgeGender); 
       return output.AsReadOnly(); 

       //return obj.newTest(s); 
      } 
      catch (Exception ex) 
      { 
       return null; 

      } 



     } 
    } 
}

另外：

注册后。净SDK库和USQL

提交作业提到他们，显示下面的文本在输出

System.InvalidOperationException: Collection was modified; enumeration operation may not execute. 
    at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) 
    at System.Collections.Generic.List`1.Enumerator.MoveNextRare() 
    at System.Collections.Generic.List`1.Enumerator.MoveNext() 
    at Microsoft.Cosmos.ScopeStudio.VsExtension.ProjectSystem.ScopeProjectNode.get_ReferenceInfoList() 
    at Microsoft.Cosmos.ScopeStudio.BusinessObjects.Common.ScriptToProjectTable.GetProjectReferenceList(String scriptFilePath) 
    at Microsoft.Cosmos.ScopeStudio.UserInterface.SQLIP.BaseSubmissionViewModel`1.GetScriptContentsWithReference(ProductFunctionType productType) 
    at Microsoft.Cosmos.ScopeStudio.UserInterface.SQLIP.DataLakeJobSubmissionViewModel`1.DoJobSubmission()

添加附加参考图

注册大会ADLS

来源

2017-05-10 Harsimranjeet Singh

我有点对你试图达到的目标感到困惑。您是否尝试从U-SQL用户定义的运算符中调用Azure SDK调用？这是行不通的，因为U-SQL容器不允许调用Web服务API，包括数据湖REST API。

来源

2017-05-12 07:37:15

我猜它可以在本地运行，最终它是，但不是在ADLS上，而阅读你的答案之一https://social.msdn.microsoft.com/Forums/en-US/01a3a359-4f13-4535-8a59-e32acbdcf77e/from-ac-usql-class-assembly-make-an-external-http-web-request-to-get-geo-location-from-ip？forum = AzureDataLake –

所以没有可能，如果我存储了我的映像路径在ADLA表中，可以用值“/sample/image/image1.jpg”来说，并且希望使用相同的路径将它移动到ADLS中的其他文件夹，可以说“/Customer/image/image1.jpg”。或者如果我需要该路径的图像作为认知服务（FACE API）的参数。 –

本地运行未在容器中运行，因此它不会像服务器容器那样阻塞您。此外，您的本地计算机还有许多其他选项可用于启动DDOS，因此阻止您执行此操作并不重要;）。您可以用许多不同的方式复制数据。我仍然不是100％你想做什么，但是假设你想在OUTPUT语句目标中使用一个表内的路径，你将不得不编写一个脚本来生成包含OUTPUT的脚本，然后运行生成的脚本。 –

您想使用PROCESS表达式（https://msdn.microsoft.com/library/en-us/Mt621322.aspx）。处理器可以生成零个或一个输出行。

另外：

此外，你需要注册相关的模块，组件，如引用它们在你的U-SQL脚本（参见https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-u-sql-programmability-guide#register-u-sql-assemblies）：

REFERENCE ASSEMBLY [Microsoft.Azure.Management.DataLake.Store]; 
REFERENCE ASSEMBLY [Microsoft.Azure.Management.DataLake.StoreUploader]; 
REFERENCE ASSEMBLY [Microsoft.IdentityModel.Clients.ActiveDirectory]; 
REFERENCE ASSEMBLY [Microsoft.Rest.ClientRuntime]; 
REFERENCE ASSEMBLY [Microsoft.Rest.ClientRuntime.Azure]; 
REFERENCE ASSEMBLY [Microsoft.Rest.ClientRuntime.Azure.Authentication]; 
REFERENCE ASSEMBLY [Newtonsoft.Json];

来源

2017-05-10 15:20:39

我刚刚更新了帖子，尝试PROCESS表达式后仍然面临问题，任何建议亚历山大，关于我在这里失踪。 –

你能展示完整的例外吗？你有一个MissingMethodException，你是否缺少一些DLLs是你的程序集？ –

获取输出痛苦的错误，同时建议您执行其他步骤，同时更新屏幕截图 –

我们不能使用USQL Custom Code和usql上传文档/图像吗？

回答

相关问题